Bump DefaultKubeBinaryVersion to 1.33 #128279

Jefftree · 2024-10-23T01:30:37Z

What type of PR is this?

/kind cleanup

What this PR does / why we need it:

Bump DefaultKubeBinaryVersion to 1.33

Fixed the etcd tests to account for previous emulated versions
Removed a emulated version test that tests for 1.30. Unfortunately we don't have a GA API in the 1.31-1.32 range to test the storage version, it might be worthwhile to (similar to the metrics test) create a fake API that we inject into tests for this purpose.
Changed the version validation to support n-3 with clamp at 1.31 min for emulatedVersion and 1.30 min for minCompatibilityVersion

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

/triage accepted

test/integration/etcd/data.go

Jefftree

Updated PTAL

test/integration/apiserver/apiserver_test.go

test/integration/metrics/metrics_test.go

Jefftree · 2024-12-20T18:08:53Z

staging/src/k8s.io/component-base/featuregate/registry_test.go

@@ -118,7 +118,7 @@ func TestVersionFlagOptions(t *testing.T) {
 func TestVersionFlagOptionsWithMapping(t *testing.T) {
 	r := testRegistry(t)
 	utilruntime.Must(r.SetEmulationVersionMapping(testComponent, DefaultKubeComponent,
-		func(from *version.Version) *version.Version { return from.OffsetMinor(3) }))
+		func(from *version.Version) *version.Version { return version.MajorMinor(1, from.Minor()+23) }))


This was updated because the original test was not written to check the actual values of the mapping, just a placeholder for a mapping exist. So it is actually mapping test 2.8 to kube 2.11, which is not aligned with the kube binary at all.

This change fixes the test to the correct version mapping.

cc @siyuanfoundation @liggitt

is adding 23 to the minor correct?

Mapping is from test component 2.7 -> kube 1.30 and 2.8 -> kube 1.31 which is where the +23 was derived from specifically for this test.

test/integration/etcd/server.go

staging/src/k8s.io/component-base/version/version.go

test/integration/etcd/etcd_storage_path_test.go

Jefftree · 2024-12-20T20:22:29Z

k8s.io/client-go/tools: cache expand_less | 44s
{Failed  === RUN   TestAddWhileActive
polling
==================
WARNING: DATA RACE

This seems a bit worrisome but I was also able to reproduce this on master. Created #129352

/test pull-kubernetes-unit

liggitt · 2024-12-20T23:17:31Z

/lgtm
/approve

k8s-ci-robot · 2024-12-20T23:17:38Z

LGTM label has been added.

Git tree hash: 6d8358d261407306af388a95732946346359e891

k8s-ci-robot · 2024-12-20T23:17:46Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Jefftree, liggitt

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~staging/src/k8s.io/apiserver/pkg/server/options/OWNERS~~ [liggitt]
~~staging/src/k8s.io/component-base/featuregate/OWNERS~~ [liggitt]
~~staging/src/k8s.io/component-base/version/OWNERS~~ [liggitt]
~~test/OWNERS~~ [liggitt]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

aojea · 2025-01-09T00:26:07Z

test/integration/etcd/etcd_storage_path_test.go

+		// Only test for beta and GA APIs with emulated version.
+		featuregatetesting.SetFeatureGateEmulationVersionDuringTest(t, feature.DefaultFeatureGate, version.MustParse(v))
+		featuregatetesting.SetFeatureGateDuringTest(t, feature.DefaultFeatureGate, "AllBeta", true)
+		registerEffectiveEmulationVersion(t)


This presents a problem , and it took me several days to find it but I still can't figure out how to solve it.

If you checkout my PR #128971 and apply the following patch

diff --git a/pkg/controlplane/apiserver/apis.go b/pkg/controlplane/apiserver/apis.go index 7ed3a58489b..47dc2cf2467 100644 --- a/pkg/controlplane/apiserver/apis.go +++ b/pkg/controlplane/apiserver/apis.go @@ -89,6 +89,7 @@ func (s *Server) InstallAPIs(restStorageProviders ...RESTStorageProvider) error nonLegacy := []*genericapiserver.APIGroupInfo{} // used later in the loop to filter the served resource by those that have expired. + klog.Infof("DEBUG ---------- effective emulation version %s", s.GenericAPIServer.EffectiveVersion.EmulationVersion().String()) resourceExpirationEvaluator, err := genericapiserver.NewResourceExpirationEvaluator(s.GenericAPIServer.EffectiveVersion.EmulationVersion()) if err != nil { return err

and runs the test

go test -timeout 600s -run ^TestEtcdStoragePath$ k8s.io/kubernetes/test/integration/etcd -v -count 1 | grep -E 'DEBUG|TestEtcdStoragePath'

you can see that the 1.33 fails because it it keeps using the emulated version 1.32 with 1.33 , this causes the APIExpiration thingy to disable the v1 version of the apiserver for servicecidrs and ipaddresses, and the cluster can not start

=== RUN TestEtcdStoragePath/1.33 I0109 00:21:01.588321 2218629 serving.go:380] Generated self-signed cert (/tmp/TestEtcdStoragePath1.3382862738/apiserver.crt, /tmp/TestEtcdStoragePath1.3382862738/apiserver.key) I0109 00:21:01.590392 2218629 dynamic_serving_content.go:116] "Loaded a new cert/key pair" name="serving-cert::/tmp/TestEtcdStoragePath1.3382862738/apiserver.crt::/tmp/TestEtcdStoragePath1.3382862738/apiserver.key" I0109 00:21:01.989808 2218629 apf_controller.go:292] NewTestableController "Controller" with serverConcurrencyLimit=600, name=Controller, asFieldManager="api-priority-and-fairness-config-consumer-v1" I0109 00:21:02.018277 2218629 apis.go:92] DEBUG ---------- effective emulation version 1.32

however, if you run the test individually it works fine

PASS {"level":"warn","ts":"2025-01-09T00:24:46.595770Z","caller":"embed/serve.go:160","msg":"stopping insecure grpc server due to error","error":"accept tcp 127.0.0.1:42889: use of closed network connection"} {"level":"warn","ts":"2025-01-09T00:24:46.595848Z","caller":"embed/serve.go:162","msg":"stopped insecure grpc server due to error","error":"accept tcp 127.0.0.1:42889: use of closed network connection"} I0109 00:24:51.596565 2223139 etcd.go:158] etcd didn't exit in 5 seconds, killing it I0109 00:24:51.596601 2223139 etcd.go:164] etcd exit status: signal: terminated ok k8s.io/kubernetes/test/integration/etcd 19.938s

cc: @liggitt @Jefftree @siyuanfoundation

Confirming this is an issue. The variable is actually reset properly but we need to call https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-apiserver/app/testing/testserver.go#L208-L209 to register the proper effective version with the component. Will send a fix

Sorry about the frustration. #129416 should be able to fix the problem.

Sorry about the frustration. #129416 should be able to fix the problem.

not sorry about nothing, these things happen, we need to break some eggs to do the omelette :)

Jefftree force-pushed the compat-133 branch from ba324c8 to 9e050a7 Compare October 23, 2024 01:30

k8s-ci-robot requested review from jpbetz and Verolop October 23, 2024 01:30

Jefftree force-pushed the compat-133 branch from 9e050a7 to 854a83a Compare November 8, 2024 16:13

Jefftree mentioned this pull request Nov 8, 2024

Emulation version: Remove hard coded DefaultKubeBinaryVersion value. #126686

Open

Jefftree force-pushed the compat-133 branch from aaff5bd to 629f820 Compare December 5, 2024 19:56

Jefftree force-pushed the compat-133 branch from bee28fc to a8351a0 Compare December 10, 2024 20:04

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Dec 10, 2024

Jefftree commented Dec 12, 2024

View reviewed changes

test/integration/etcd/data.go Outdated Show resolved Hide resolved

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Dec 20, 2024

Jefftree force-pushed the compat-133 branch 3 times, most recently from fa38c5f to bb10eaa Compare December 20, 2024 18:04

Jefftree commented Dec 20, 2024

View reviewed changes

test/integration/etcd/server.go Show resolved Hide resolved

Jefftree force-pushed the compat-133 branch from bb10eaa to c62628e Compare December 20, 2024 18:16

liggitt reviewed Dec 20, 2024

View reviewed changes

staging/src/k8s.io/component-base/version/version.go Show resolved Hide resolved

liggitt reviewed Dec 20, 2024

View reviewed changes

test/integration/etcd/etcd_storage_path_test.go Outdated Show resolved Hide resolved

liggitt reviewed Dec 20, 2024

View reviewed changes

test/integration/etcd/etcd_storage_path_test.go Show resolved Hide resolved

Jefftree force-pushed the compat-133 branch from 8b0f804 to c689027 Compare December 20, 2024 19:06

Jefftree added 5 commits December 20, 2024 19:07

Bump binary version to 1.33

4e8477b

Expand emulated version range to n-3 with 1.31 floor

cf28c04

remove emulated version test for un-emulatable 1.30

dc476e9

expand etcd storage tests to test for 1.31..1.33

6c94adc

fix metrics test

59fcd7b

Jefftree force-pushed the compat-133 branch from c689027 to 59fcd7b Compare December 20, 2024 19:08

Jefftree mentioned this pull request Dec 20, 2024

Properly register emulation version in etcd testing #129308

Closed

k8s-ci-robot assigned liggitt Dec 20, 2024

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Dec 20, 2024

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 20, 2024

k8s-ci-robot merged commit 9d82148 into kubernetes:master Dec 20, 2024
14 checks passed

k8s-ci-robot added this to the v1.33 milestone Dec 20, 2024

Jefftree mentioned this pull request Jan 6, 2025

etcd compatibility fixtures should derive version rather than hard code when APIs were introduced/removed #129491

Open

aojea reviewed Jan 9, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bump DefaultKubeBinaryVersion to 1.33 #128279

Bump DefaultKubeBinaryVersion to 1.33 #128279

Jefftree commented Oct 23, 2024 •

edited

Loading

Jefftree left a comment

Jefftree Dec 20, 2024

liggitt Dec 20, 2024

Jefftree Dec 20, 2024

Jefftree commented Dec 20, 2024 •

edited

Loading

liggitt commented Dec 20, 2024

k8s-ci-robot commented Dec 20, 2024

k8s-ci-robot commented Dec 20, 2024

aojea Jan 9, 2025 •

edited

Loading

Jefftree Jan 9, 2025

siyuanfoundation Jan 9, 2025

aojea Jan 9, 2025

Bump DefaultKubeBinaryVersion to 1.33 #128279

Bump DefaultKubeBinaryVersion to 1.33 #128279

Conversation

Jefftree commented Oct 23, 2024 • edited Loading

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

Jefftree left a comment

Choose a reason for hiding this comment

Jefftree Dec 20, 2024

Choose a reason for hiding this comment

liggitt Dec 20, 2024

Choose a reason for hiding this comment

Jefftree Dec 20, 2024

Choose a reason for hiding this comment

Jefftree commented Dec 20, 2024 • edited Loading

liggitt commented Dec 20, 2024

k8s-ci-robot commented Dec 20, 2024

k8s-ci-robot commented Dec 20, 2024

aojea Jan 9, 2025 • edited Loading

Choose a reason for hiding this comment

Jefftree Jan 9, 2025

Choose a reason for hiding this comment

siyuanfoundation Jan 9, 2025

Choose a reason for hiding this comment

aojea Jan 9, 2025

Choose a reason for hiding this comment

Jefftree commented Oct 23, 2024 •

edited

Loading

Jefftree commented Dec 20, 2024 •

edited

Loading

aojea Jan 9, 2025 •

edited

Loading