• Release Signoff Checklist
• Summary
• Motivation
  • Goals
  • Non-Goals
• Proposal
  • User Stories (Optional)
    • Story 1
    • Story 2
  • Notes/Constraints/Caveats (Optional)
  • Risks and Mitigations
• Design Details
  • CI Jobs:
    • Scenarios:
    • Managing kubetest2 dependency
  • Test Plan
  • Graduation Criteria
  • Upgrade / Downgrade Strategy
  • Version Skew Strategy
• Production Readiness Review Questionnaire
  • Feature Enablement and Rollback
  • Rollout, Upgrade and Rollback Planning
  • Monitoring Requirements
  • Dependencies
  • Scalability
  • Troubleshooting
• Implementation History
• Drawbacks
• Alternatives
• Infrastructure Needed (Optional)

Items marked with (R) are required prior to targeting to a milestone / release.

• (R) Enhancement issue in release milestone, which links to KEP dir in kubernetes/enhancements (not the initial KEP PR)
• (R) KEP approvers have approved the KEP status as implementable
• (R) Design details are appropriately documented
• (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input
• (R) Graduation criteria is in place
• (R) Production readiness review completed
• (R) Production readiness review approved
• "Implementation History" section is up-to-date for milestone
• User-facing documentation has been created in kubernetes/website, for publication to kubernetes.io
• Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes

Kubernetes currently uses Kubetest as the interface for launching and running e2e tests. It also uses a set of scripts called “scenarios” for running common use cases such as setting up end-to-end tests or pushing CI builds.
Kubetest and scenarios have both been deprecated, (2) for a while, with a recommendation to move jobs to kubetest2.

This KEP proposes a plan for migration/deprecation of CI jobs that are in the critical path of a release.

• Converge towards a single community-supported e2e launcher tool for kubernetes
• Ensure all kubernetes/kubernetes release-blocking and merge-blocking jobs use the same e2e test tool, to reduce community maintenance burden, increase community troubleshooting awareness
• Document guides for migrating jobs away from kubetest to kubetest2 to enable self-service migration of non-blocking jobs
• Accelerate the migration from bootstrap.py to pod-utilities since Kubetest2 does not depend on bootstrap
• Enable SIG Release to deprecate legacy push-build.sh

• Implementing kubetest2 deployers for all existing kubetest providers
  • kubetest2 enables implementing custom deployers out-of-tree so replacing all of the deployers upstream is not a hard requirement
• Migrate all 1700+ e2e jobs away from the deprecated e2e test tools (e.g. scenarios, bootstrap, kubetes)
  • This KEP only covers migrating all kubernetes/kubernetes release-blocking and merge-blocking jobs.
  • We propose enforcing no further changes to the remaining deprecated tools to incentivize community migration and improvements to kubetest2.
  • Other community efforts are already underway (e.g. removing the kubernetes_build scenario)
• Backporting changes to older release branches
  • In general, no changes should be made to the release branches or their CI and the old jobs will age out.
• Decoupling from kubekins to a more minimal image
  • The kubekins image has become a kitchen-sink over the years, used by more jobs than necessary, and containing more dependencies than required for effective kubetest2 usage
  • Migrating kubekins away from bootstrap as a base image is only feasible once all jobs have migrated away from bootstrap, which is a non-goal of this KEP

For the development (main/master) branch only, NOT the existing release branches:

• Create shadow canary jobs for each presubmit-kubernetes-blocking, release-blocking to use kubetest2 as optional, non-blocking
• Monitor these canary jobs till they are stable
• Make kubetest2 jobs non-optional, non-blocking
• Make kubetest2 jobs non-optional, blocking

No changes should be made to the release branches or their CI.

• Stage 1 of shipping the kubetest2 binary involves keeping HEAD at kubetest2 unbroken (same as kubetest currently).
• Stage 2 of the proposal involves monitoring a job to become stable which might encounter discrepancies between its existing kubetest2 counterpart.

The following deployers/testers are used as part of the jobs in consideration:

• KIND:

  KIND jobs neither uses scenarios nor kubetest so most KIND jobs are noops.

  • KIND is already a supported kubetest2 deployer for basic end-to-end testing.
  • Possible future consideration for kubetest2 kind jobs:
    • A typical kind job runs using the krte image, downloads the latest kind binary, sets up the environment for the test job using e2e-k8s.sh
    • Future additions required to the KIND deployer
      • support fetching latest kind binaries
      • move over functionality from e2e-k8s.sh as part of the kind deployer

• GCE:

  GCE jobs are the ones that use scenarios and kubetest the most. Most of the GCE deployer logic has been ported over to kubetest2, see implementation history for details. We also have a canary job for GCE conformance using kubetest2.

• Node:

  Most of the node e2e logic is consolidated as part of the e2e test themselves. So the jobs basically create noop clusters which will need to be implemented in kubetest2. This noop deployer will also be useful for other build jobs. Additionally, we can have a short wrapper over the noop deployer specifically for the node deployer to mimic what kubetest does, e.g. setting up GCP SSH Keys

• Scale:

  Scale tests use the clusterloader2 framework in kubernetes/perf-tests. kubetest2 already has support for clusterloader2 tester.

• Ginkgo:

  All of the jobs (apart from scale jobs) are using the kubernetes test/e2e framework which uses ginkgo for the end-to-end tests. kubetest2 already supports this ginkgo tester. Some tests require checking for version-skew between the client and server version, support for which will need to be added in kubetest2.

There are mainly 3 scenarios that are currently used:

• scenario=execute is used to run arbitrary test scripts. It can be replaced by the kubetest2 exec tester.

  kubetest2 --test=exec – script.sh

• scenario=kubernetes_e2e is used for setting up the environment for end-to-end jobs.

  • It specifies it’s own set of flags to customize the setup all of which are deprecated and unused.
  • It has a lot of functionality for setting up Kops clusters which will need to be ported over to be part of the kubetest2 kops deployer. They are currently being monitored in testgrid here: https://testgrid.k8s.io/kops-kubetest2#e2e-kubetest2

  Most other usages in jobs are used to indirectly invoke kubetest, which will be replaced by their equivalent kubetest2 commands.

  kubetest2 <deployer> --build --up --down --deployer-specific-setup-flags

• scenarios=kubernetes_build which is primarily used for pushing CI builds to GCS. This is where we are also using the legacy push-build scripts (in addition to kubetest --stage). These will be replaced by KREL.

  Optionally they can also be migrated to use the kubetest2 KREL integration.

  kubetest2 noop --build --stage=gs://bucket

  Additionally, there is exactly one usage of scenario=canarypush which can be replaced by an equivalent script and migrated to using kubetest2 exec tester.

• Add a script to kubekins-e2e to fetch pre-built latest kubetest2 binaries (stored in GCS) as part of each job
• Distribute a tagged version of kubetest2 binary for each release and add to variants.yaml for kubekins-e2e
• Publish a kubetest2 specific images (optional)

• This will be declared alpha when:

  • Jobs fetch latest pre-built kubetest2 binaries at runtime
  • GCE jobs have corresponding kubetest2 shadow jobs on k/k master
  • Noop support is added for node deployer and build

• This will be declared beta when:

  • A guide for migration from kubetest to kubetest2 exists
  • GCE jobs are monitored to be stable, and have moved over to kubetest2
  • Node, Scale jobs have corresponding shadow jobs on k/k master

• This will be declared GA when:

  • Jobs use pre-installed tagged versions of kubetest2 binaries
  • Tracking issues exist for each SIG to migrate their jobs from kubetest to kubetest2
  • All kubernetes/kubernetes master presubmit-kubernetes-blocking, release-blocking jobs are using kubetest2

n/a (this is a build/test change for the kubernetes project, maintains parity with existing merge-blocking and release-blocking jobs)

n/a (this is a build/test change for the kubernetes project, maintains parity with existing merge-blocking and release-blocking jobs)

n/a (this is a build/test change for the kubernetes project, not relevant to monitoring production clusters)

n/a (this is a build/test change for the kubernetes project, maintains parity with existing merge-blocking and release-blocking jobs)

n/a (this is a build/test change for the kubernetes project, maintains parity with existing merge-blocking and release-blocking jobs, including scalability jobs)

n/a (this is a build/test change for the kubernetes project, not relevant to troubleshooting problems in production cluster)

• 2020-07-07: Kind deployer support added in kubetest2
• 2020-07-15: Boskos support is added to kubetest2
• 2020-08-10: GCE deployer support added in kubetest2. see also this document for the implementation details
• 2020-07-30: Ginkgo tester support added in kubetest2
• 2020-09-14: Clusterloader2 tester support added in kubetest2

• Continue using kubetest, scenarios, push-build.sh as-is

  This has several drawbacks:

  • Kubetest lack proper owners and active contribution. Support is mostly added as bugs are discovered.
  • Scenarios are mostly legacy hacky scripts, which will provide better maintainability if it is migrated to a proper framework.
  • Push-build.sh is another legacy script which SIG Release is actively moving away from with efforts such as KREL, Anago to be rewritten in go.

• Improve kubetest, scenarios, push-build.sh

  This has several drawbacks:

  • Kubetest lacks a good separation of concerns which leads to changes having a large blast radius. See: why kubetest2
  • Most scenarios are obsolete and have only trivial usages all of which can be handled by kubetest2 through a single entrypoint.
  • push-build.sh is written in bash which is hard to maintain and provides no additional value over it’s go equivalent KREL push.