Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: prefetch shuttle-service crates #461

Merged
merged 2 commits into from
Nov 4, 2022
Merged

feat: prefetch shuttle-service crates #461

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d38dae5
feat: prefetch shuttle-service crates
chesedo Nov 3, 2022
1f0e35c
refactor: add comment to prepare.sh files
chesedo Nov 4, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 6 additions & 12 deletions Containerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,8 @@ FROM shuttle-build AS builder
COPY --from=planner /build/recipe.json recipe.json
RUN cargo chef cook --recipe-path recipe.json
COPY --from=cache /build .
ARG crate
RUN cargo build --bin ${crate}
ARG folder
RUN cargo build --bin shuttle-${folder}

FROM rust:1.63.0-buster as shuttle-common
RUN apt-get update &&\
Expand All @@ -28,14 +28,8 @@ RUN rustup component add rust-src
COPY --from=cache /build/ /usr/src/shuttle/

FROM shuttle-common
ARG crate
SHELL ["/bin/bash", "-c"]
RUN mkdir -p $CARGO_HOME; \
echo $'[patch.crates-io] \n\
shuttle-service = { path = "/usr/src/shuttle/service" } \n\
shuttle-aws-rds = { path = "/usr/src/shuttle/resources/aws-rds" } \n\
shuttle-persist = { path = "/usr/src/shuttle/resources/persist" } \n\
shuttle-shared-db = { path = "/usr/src/shuttle/resources/shared-db" } \n\
shuttle-secrets = { path = "/usr/src/shuttle/resources/secrets" }' > $CARGO_HOME/config.toml
COPY --from=builder /build/target/debug/${crate} /usr/local/bin/service
ARG folder
COPY ${folder}/prepare.sh /prepare.sh
RUN /prepare.sh
COPY --from=builder /build/target/debug/shuttle-${folder} /usr/local/bin/service
brokad marked this conversation as resolved.
Show resolved Hide resolved
ENTRYPOINT ["/usr/local/bin/service"]
2 changes: 1 addition & 1 deletion Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ down: docker-compose.rendered.yml

shuttle-%: ${SRC} Cargo.lock
docker buildx build \
--build-arg crate=shuttle-$(*) \
--build-arg folder=$(*) \
--tag $(CONTAINER_REGISTRY)/$(*):$(COMMIT_SHA) \
--tag $(CONTAINER_REGISTRY)/$(*):$(TAG) \
--tag $(CONTAINER_REGISTRY)/$(*):latest \
Expand Down
14 changes: 14 additions & 0 deletions deployer/prepare.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
#!/usr/bin/env sh
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add a quick note in these files to say where they're used (i.e. ../Containerfile) for future maintainers?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was hoping you would have a better/cleaner idea for how to do this? 😅

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Else, I'll just comment these properly

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Extra comments added

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was hoping you would have a better/cleaner idea for how to do this? 😅

Not really I'm afraid! The only things I can think of are:

  • Just add RUN cargo fetch in the shuttle-common image building step. It does mean the base layer gets a bit bigger, but should not impact things too much since the size difference can't be immense. Or:
  • have this be an "init" step of deployer, triggered from there

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For interest sake, this increases the deployer container size from 2Gb to 3Gb and the compile speedup is quite huge on my system


# Patch crates to be on same versions
mkdir -p $CARGO_HOME; \
echo '[patch.crates-io]
chesedo marked this conversation as resolved.
Show resolved Hide resolved
shuttle-service = { path = "/usr/src/shuttle/service" }
shuttle-aws-rds = { path = "/usr/src/shuttle/resources/aws-rds" }
shuttle-persist = { path = "/usr/src/shuttle/resources/persist" }
shuttle-shared-db = { path = "/usr/src/shuttle/resources/shared-db" }
shuttle-secrets = { path = "/usr/src/shuttle/resources/secrets" }' > $CARGO_HOME/config.toml

# Prefetch crates.io index
cd /usr/src/shuttle/service
cargo fetch
3 changes: 3 additions & 0 deletions gateway/prepare.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
#!/usr/bin/env sh

# Nothing to prepare in container image here
3 changes: 3 additions & 0 deletions provisioner/prepare.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
#!/usr/bin/env sh

# Nothing to prepare in container image here