@spxtr Can you take a look? I think the gke build has the same problem.

Looks like they've gotten significantly slower since the last time this came up. I'll try and figure out why.

This happened again and is continuing to block the merge queue. @spxtr can you make a bandaid that extends the timeout?

Done, but I need to stress that we need to know why the duration is climbing. I'm afraid in a few months I'll get another issue that says "kubernetes-test-go is timing out".

Thanks. Yes, we can leave this issue open until we figure it out.

@spxtr maybe another bandaid for our build? http://kubekins.dls.corp.google.com/view/Critical%20Builds/job/kubernetes-build/9413/

Actually, it was taking 14-17 mins, so 30 minute timeout should hopefully be enough. hm.

The build is the result of a PR going in that does affect the build. It had to download a new build image, and it also needs to build more tarballs. We might need to legitimately bump that one's timeout.

It looks like it's about to time out again.

Barely passed. The build got slower by a factor of two after #23931 went in. This was somewhat expected. I'll bump the timeout.

We should exclude those platforms in our testing, maybe? We don't need ppc or arm binaries. Should we roll back #23931?

I don't think we should roll it back. We want kubernetes-build to do a full release build, which now includes building for those architectures.

However, if the PR Jenkins e2e job starts timing out, then we should probably change it to do a quick-release instead of the full release.

cc @luxas

Dropped the priority now that the bleeding has stopped. I'll try to figure out why the test-go job got slower.

@luxas this is a fairly significant increase in build and release times with #23931. Is there anything we can do to mitigate the increase? I see many packages downloaded during the build. Are we maybe downloading more than we need to? Can we cache anything somehow/somewhere? Can we parallelize package updating and/or building?

The problem is not all the things that get downloaded into kube-cross, that's just one time.
It's the building time that's a small problem. Since go1.5+ building may get ~2x slower, and now we've both upgraded to go1.6 and added more server platforms, so the increase in build time is expected. Sorry for not notifying you in beforehand though.

Steps we could take to decrease the time:

• Move cmd/linkcheck to test targets. I don't know why that one is considered a server target.
• Build test targets only for linux/amd64 (now linux/amd64, windows/amd64, darwin/amd64, linux/arm)
• Remove addon images: don't ship kube-registry-proxy and pause images in tars. #23605
• Consider dropping support for */386 for kubectl
• Remove cmd/kubemark for arm, arm64 and ppc64le. I don't think it's required from official builds.

@spxtr We should do a full build on every CI run for all arches, so we may detect regressions in the code

Move cmd/linkcheck to test targets. I don't know why that one is considered a server target.

Whoops, I meant to make a PR to do that a while ago. cc @caesarxuchao

@spxtr thanks for letting me know. Do you want me to send the PR or will you?

Build test targets only for linux/amd64 (now linux/amd64, windows/amd64, darwin/amd64, linux/arm)

I think we definitely want to build test targets for at least darwin, since plenty of people develop on a mac. It might be worth dropping tests for arm and windows.

Remove cmd/kubemark for arm, arm64 and ppc64le. I don't think it's required from official builds.

sgtm

All of our kubernetes-build and kubernetes-test-go jobs are running on a single n1-highmem-32 instance, which is hitting 100% CPU usage fairly often. This is most likely why our test-go times are so inconsistent lately.

Is there anything I may help with? (Not having access to your servers)

@luxas I think I can handle the test-go problems. Feel free to work on your other suggestions, I think they're good. Thanks, though :)

The timeout is too aggressive:

http://kubekins.dls.corp.google.com/job/kubernetes-test-go/buildTimeTrend shows a passing run in 75 minutes (11026) and the average passing run time is in the mid 50m. We need at least a 100m timeout (2x average 50m runtime) rather than 80.

Or to put it another way: in order to run this job reliably I want the timeout to be twice the average runtime, rather than a couple additional minutes.

@fejta I'm wary of a 100 minute timeout. Last time this came up the job was taking ~35 minutes (#23127) so an 80 minute timeout was fine. Some time last week it started taking 60-80 minutes and occasionally timing out. I'd like to know why. In the meantime, to fix the submit queue lets bump the timeout.

I'm also starting to think we should run verify-*.sh, unit/integration tests, and test-cmd.sh in separate Jenkins jobs. kubernetes-test-go blocks the queue for over an hour now.