Containerize verification checks, unit tests, and integration test #14781
Description
In the pull request Jenkins, we're running up to 8 instances of verification checks, build, unit tests, integration tests, and e2e tests.
Unfortunately, it seems like several of these components were not designed to run simultaneous to other tests on the same machine.
In particular, several unit tests seem to write to static file names in /tmp (or equivalent), which is a fairly easy fix. Some of the verification checks and the integration test start up etcd, however, and the etcd helper checks that only a single etcd binary is running on the host (and uses a static port for it).
For example:
Verifying ./hack/../hack/verify-generated-swagger-docs.sh
+++ [0929 12:56:12] Building go targets for linux/amd64:
cmd/genswaggertypedocs
+++ [0929 12:56:12] Placing binaries
Generating swagger type docs for unversioned
Generating swagger type docs for v1
Generating swagger type docs for experimental/v1alpha1
+++ [0929 12:56:15] Building go targets for linux/amd64:
cmd/kube-apiserver
+++ [0929 12:56:28] Placing binaries
etcd appears to already be running on this machine (7295 etcd) (or its a zombie and you need to kill its parent).
retry after you resolve this etcd error.
+++ [0929 12:56:29] Clean up complete
!!! Error in /jenkins-master-data/jobs/kubernetes-pull-build-test-e2e-gce/workspace@5/hack/update-swagger-spec.sh:31
'"${KUBE_ROOT}/hack/after-build/update-swagger-spec.sh" "$@"' exited with status 1
Call stack:
1: /jenkins-master-data/jobs/kubernetes-pull-build-test-e2e-gce/workspace@5/hack/update-swagger-spec.sh:31 main(...)
Exiting with status 1
!!! Error in /jenkins-master-data/jobs/kubernetes-pull-build-test-e2e-gce/workspace@5/hack/update-generated-swagger-docs.sh:68
'"${KUBE_ROOT}/hack/update-swagger-spec.sh"' exited with status 1
Call stack:
1: /jenkins-master-data/jobs/kubernetes-pull-build-test-e2e-gce/workspace@5/hack/update-generated-swagger-docs.sh:68 main(...)
Exiting with status 1
!!! Error in /jenkins-master-data/jobs/kubernetes-pull-build-test-e2e-gce/workspace@5/hack/after-build/verify-generated-swagger-docs.sh:46
'"${KUBE_ROOT}/hack/update-generated-swagger-docs.sh"' exited with status 1
Call stack:
1: /jenkins-master-data/jobs/kubernetes-pull-build-test-e2e-gce/workspace@5/hack/after-build/verify-generated-swagger-docs.sh:46 main(...)
Exiting with status 1
!!! Error in ./hack/../hack/verify-generated-swagger-docs.sh:28
'"${KUBE_ROOT}/hack/after-build/verify-generated-swagger-docs.sh" "$@"' exited with status 1
Call stack:
1: ./hack/../hack/verify-generated-swagger-docs.sh:28 main(...)
Exiting with status 1
FAILED
+++ [0929 13:03:14] Building go targets for linux/amd64:
cmd/integration
+++ [0929 13:03:31] Placing binaries
etcd appears to already be running on this machine (7295 etcd) (or its a zombie and you need to kill its parent).
retry after you resolve this etcd error.
+++ [0929 13:03:32] Integration test cleanup complete
We might be able to fix the etcd issue separately, or we can just figure out how to containerize all of these tests so they can't possibly interfere with each other.
Note that we're ignoring the exit status of the verification checks and unit/integration tests right now, so these etcd failures are effectively being ignored (though it is causing us to miss test coverage).
@kubernetes/goog-testing