roachtest/ci: build all binaries in a separate job #139790
Open
Description
With the addition of FIPS for nightly runs [1], the time to build all required binaries has reached nearly 1 hour,
[03:26:42]W: [Step 2/2] + /go/src/github.com/cockroachdb/cockroach/build/teamcity/cockroach/nightlies/roachtest_compile_bits.sh amd64
...
[03:48:00]W: [Step 2/2] + /go/src/github.com/cockroachdb/cockroach/build/teamcity/cockroach/nightlies/roachtest_compile_bits.sh amd64-fips
...
[04:03:32]W: [Step 2/2] + /go/src/github.com/cockroachdb/cockroach/build/teamcity/cockroach/nightlies/roachtest_compile_bits.sh arm64
...
[04:20:12] : [Step 2/2] -rwxrwxrwx 1 roach roach 305544704 Jan 23 03:37 cockroach-ea.linux-amd64^M
[04:20:12] : [Step 2/2] -rwxrwxrwx 1 roach roach 306312960 Jan 23 03:55 cockroach-ea.linux-amd64-fips^M
[04:20:12] : [Step 2/2] -rwxrwxrwx 1 roach roach 292437408 Jan 23 04:14 cockroach-ea.linux-arm64^M
[04:20:12] : [Step 2/2] -rwxrwxrwx 1 roach roach 305026800 Jan 23 03:34 cockroach.linux-amd64^M
[04:20:12] : [Step 2/2] -rwxrwxrwx 1 roach roach 305796384 Jan 23 03:52 cockroach.linux-amd64-fips^M
[04:20:12] : [Step 2/2] -rwxrwxrwx 1 roach roach 292007872 Jan 23 04:11 cockroach.linux-arm64^M
[04:20:12] : [Step 2/2] -rwxrwxrwx 1 roach roach 136019576 Jan 23 04:05 roachprod^M
[04:20:12] : [Step 2/2] -rwxrwxrwx 1 roach roach 136019576 Jan 23 04:05 roachprod.linux-amd64^M
[04:20:12] : [Step 2/2] -rwxrwxrwx 1 roach roach 349449696 Jan 23 04:06 roachtest^M
[04:20:12] : [Step 2/2] -rwxrwxrwx 1 roach roach 349449696 Jan 23 04:06 roachtest.linux-amd64^M
[04:20:12] : [Step 2/2] -rwxrwxrwx 1 roach roach 149396784 Jan 23 03:48 workload.linux-amd64^M
[04:20:12] : [Step 2/2] -rwxrwxrwx 1 roach roach 150147520 Jan 23 03:57 workload.linux-amd64-fips^M
[04:20:12] : [Step 2/2] -rwxrwxrwx 1 roach roach 145222160 Jan 23 04:20 workload.linux-arm64^M
That's essentially 2 hours wasted since both AWS and Azure need the same binaries. Thus, the binaries should be built in a separate ci job. As an added bonus, the binaries will be kept as artifacts. Occasionally, it's useful to grab the binary when debugging a specific run.
[1] #139510
Jira issue: CRDB-46813