So, I've tested the Dockerfile in most configurations imaginable: native and cross-comile builds on both ARM and x86 (on M1 Mac, through docker virtualization; takes ages :/), without lld, with lld installed and with lld enabled for Rust. However, I've mostly tested debug and no-LTO release builds, as release docker builds with "fat" LTO take literal ages (20+ minutes) on my machine.

My results:

• lld, if present, speeds up C/C++ dependencies compilation and gives a reliably observable ~20 seconds boost (though, it takes 5-10 seconds to be installed)
• if enabled for Rust, it seems to make absolutely no difference

I've currently pushed a change with lld enabled for Rust code, with a simple way to disable it. Could you guys test how does it work on your machines during the day?

• docker buildx build would build in release mode, with lld enabled
• docker buildx build --build-arg LINKER= would disable the lld
• docker buildx build --build-arg PROFILE=dev would build in debug mode
• (and, of course, you use both: docker buildx build --build-arg PROFILE=dev --build-arg LINKER=)

Note the times for builder 6/8 and builder 8/8 stages.

Oh, and BTW, is unstripped debug build for linux 900-1000 MB!?

UPD: It is. 🙈

I've currently pushed a change with lld enabled for Rust code, with a simple way to disable it. Could you guys test how does it work on your machines during the day?

* `docker buildx build` would build in **release** mode, with `lld` **enabled**

* `docker buildx build --build-arg LINKER=` would _disable_ the `lld`

* `docker buildx build --build-arg PROFILE=dev` would build in _debug_ mode

* (and, of course, you use both: `docker buildx build --build-arg PROFILE=dev --build-arg LINKER=`)

Note the times for builder 6/8 and builder 8/8 stages.

I ran the four commands at 09a5ed6, see my results below. Notice that I did add --no-cache:

I ran this on Ubuntu (Linux 5.19 x86_64), AMD Ryzen 9 5900X (12/24 cores), 32GB RAM.

I guess that for release builds there are no visible improvements (outside the margin of error).

Figured why release Docker builds failed on my Mac... it was RAM. Increased RAM limit available to Docker to 12GB and it works now. Didn't note the build time (before I've cleared the screen), though. 🤦‍♀️

Rebased to current dev. All speedups from #1856 and #1859 present.

Here's a comparison between "cold" builds. Left is this PR (aca7221, CI link), right is #1859 (d0933e5, CI link).

Screenshots

And here's a comparison between "cold" build (aca7221, CI link) and "hot" build (dbbac23, CI link) in this PR.

Screenshots

And, for the reference, "cold" build (d0933e5, CI link) and "hot" build (d4e4453, CI link) in the #1859.

Screenshots

Rebased on recent dev.

Here is my checklist:

• docker buildx works for both - amd and arm
• regular docker build . works
• Cargo uses release profile by default, but debug profile configured in CI
• Caches work
• mold is used as much as possible, but at least on amd arch
• changes in code does not invalidate dependencies cache

If those checked, then - LGTM

@generall I'll verify all of these once again (a bit) later, but unless something got accidentally broken recently, all of them should be good.

@generall I've tested the Dockerfile once again and marked most items in the checklist. I need help testing on AMD64 host (as I only have macOS, which is ARM).

I think sufficient tests would be:

• docker build .
• docker buildx build .
  • then modify any Rust file, re-run docker buildx build . and check that steps 2 to 7 of the builder stage are cached (look for the CACHED [builder 2/9] lines)
  • (it's kinda optional, but it's a quick test)
• docker buildx build . --platform linux/arm64
• (also, you better run all these steps with --build-arg PROFILE=ci to not wait for eternity for LTO to complete)

All of these works on my machine. E.g., "plain" docker build and native (aarch64 target) and cross (amd64 target) compilation with docker buildx.

@ffuugoo I'll give these a spin. I'm on Linux with amd64.

@ffuugoo All builds (build, buildx, buildx for arm64) work fine on my machine!

For cross compilation with buildx I did have to explicitly set a custom builder like --builder=container, which I created with docker buildx create --driver=docker-container --name=container. This has to do with my setup though, and don't think it can be improved upon in this PR.

Cache works with buildx.

I used --build-arg PROFILE=ci on all.