Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Revert "[FG:InPlacePodVerticalScaling] Graduate to Beta" #128875

Merged
merged 1 commit into from
Nov 20, 2024
Merged

Revert "[FG:InPlacePodVerticalScaling] Graduate to Beta" #128875

merged 1 commit into from
Nov 20, 2024
+11 −31

Conversation

pacoxu
Copy link
Member

@pacoxu pacoxu commented Nov 20, 2024 •
edited by dims
Loading

@k8s-ci-robot k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-kind Indicates a PR lacks a `kind/foo` label and requires one. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Nov 20, 2024
@k8s-ci-robot k8s-ci-robot added area/kubelet area/test sig/apps Categorizes an issue or PR as relevant to SIG Apps. sig/node Categorizes an issue or PR as relevant to SIG Node. sig/testing Categorizes an issue or PR as relevant to SIG Testing. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Nov 20, 2024
@pacoxu
Copy link
Member Author

pacoxu commented Nov 20, 2024

/test pull-kubernetes-node-kubelet-serial-containerd
/test pull-kubernetes-node-kubelet-serial-containerd-alpha-features
/test pull-kubernetes-node-kubelet-serial-containerd-kubetest2
/test pull-kubernetes-node-kubelet-serial-containerd-sidecar-containers

@pacoxu
Copy link
Member Author

pacoxu commented Nov 20, 2024

/test pull-kubernetes-e2e-capz-windows-master

@pacoxu pacoxu changed the title [WIP]Revert "[FG:InPlacePodVerticalScaling] Graduate to Beta" Revert "[FG:InPlacePodVerticalScaling] Graduate to Beta" Nov 20, 2024
@k8s-ci-robot k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. and removed do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Nov 20, 2024
@pacoxu
Copy link
Member Author

pacoxu commented Nov 20, 2024

/kind bug

@k8s-ci-robot k8s-ci-robot added kind/bug Categorizes issue or PR as related to a bug. and removed do-not-merge/needs-kind Indicates a PR lacks a `kind/foo` label and requires one. labels Nov 20, 2024
@pacoxu
Copy link
Member Author

pacoxu commented Nov 20, 2024

/cc @tallclair @SergeyKanzhelev @mrunalp @derekwaynecarr @dchen1107

@k8s-ci-robot
Copy link
Contributor

@cpanato: The provided milestone is not valid for this repository. Milestones in this repository: [next-candidate, v1.26, v1.27, v1.28, v1.29, v1.30, v1.31, v1.32, v1.33, v1.34]

Use /milestone clear to clear the milestone.

In response to this:

/milestone 1.32

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@cpanato
Copy link
Member

cpanato commented Nov 20, 2024

/milestone v1.32

@k8s-ci-robot k8s-ci-robot added this to the v1.32 milestone Nov 20, 2024
@cpanato
Copy link
Member

cpanato commented Nov 20, 2024

/triage accepted

@k8s-ci-robot k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Nov 20, 2024
@aojea
Copy link
Member

aojea commented Nov 20, 2024

weirdly, this also fixed a bunch of sig-storage tests as well in https://testgrid.k8s.io/sig-release-master-informing#capz-windows-master&width=20 (like ConfigMap should be consumable from pods in volume) ... that seems to indicate this feature enablement is doing something weird with pod / volume lifecycle ordering that impacts permissions set on volumes (even in tests where no resize changes were being made), which is really surprising

/lgtm

@k8s-ci-robot k8s-ci-robot merged commit bf70d28 into kubernetes:master Nov 20, 2024
21 of 24 checks passed
@liggitt
Copy link
Member

liggitt commented Nov 20, 2024 •
edited
Loading

Collecting issues to resolve before re-enabling this:

This was referenced Nov 20, 2024
Merged
Open
Merged
@aojea
Copy link
Member

aojea commented Nov 20, 2024

#128682 Merged on Nov 12, 8pm GMT+1

Testgrids to look at that may give signal as they seem started failing since that date, possible source for correlation

@dims
Copy link
Member

dims commented Nov 20, 2024

@aojea thanks for listing those! let's watch the upcoming runs and see what else is hiding behind this one. 🤞🏾

(I've kicked off a few runs)

@dims
Copy link
Member

dims commented Nov 20, 2024

ALL jobs @aojea pointed to above have 🟢 ✅ runs now!! thanks a ton @pacoxu

@esotsal
Copy link

esotsal commented Nov 20, 2024 •
edited
Loading

Was InPlacePodVerticalScaling Beta the issue also for the Windows CI pipeline failure?

Checking https://testgrid.k8s.io/sig-release-master-informing#capz-windows-master , why with same commit suddenly windows pipeline failed? ( last successfull / first failed )

image

@esotsal esotsal mentioned this pull request Nov 20, 2024
Open
@esotsal
Copy link

esotsal commented Nov 20, 2024 •
edited
Loading

Collecting issues to resolve before re-enabling this:

I think @liggitt it is worth adding in the list this analysis , i believe explains lot of failures seen in the pipelines ( related with tests , not InPlacePodVerticalScaling feature as is).

@esotsal
Copy link

esotsal commented Nov 21, 2024

Was InPlacePodVerticalScaling Beta the issue also for the Windows CI pipeline failure?

Answering to my self :-) , yes but not the feature it self, the tests. fyi this run from https://prow.k8s.io/pr-history/?org=kubernetes&repo=kubernetes&pr=128880 passes without the revert https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/128880/pull-kubernetes-e2e-capz-windows-master/1859388054804893696 . Retriggered to double confirm this. Seems the kubetest2 failure is the last of the three failures where we still have problems.

@pacoxu
Copy link
Member Author

pacoxu commented Nov 21, 2024

Was InPlacePodVerticalScaling Beta the issue also for the Windows CI pipeline failure?

Answering to my self :-) , yes but not the feature it self, the tests. fyi this run from https://prow.k8s.io/pr-history/?org=kubernetes&repo=kubernetes&pr=128880 passes without the revert https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/128880/pull-kubernetes-e2e-capz-windows-master/1859388054804893696 . Retriggered to double confirm this. Seems the kubetest2 failure is the last of the three failures where we still have problems.

See https://github.com/kubernetes/kubernetes/pull/128880/files#r1850861491 and the PR for more information.

The CI failure seems to be caused by the e2e pod cleanup logic, but not the feature. ┓( ´∀` )┏

@dims
Copy link
Member

dims commented Nov 21, 2024

@pacoxu unfortunately PR landed E_TOO_LATE_IN_THE_CYCLE for us to dig in like this.. so let's do this one right in 1.33.

@esotsal
Copy link

esotsal commented Nov 21, 2024

Was InPlacePodVerticalScaling Beta the issue also for the Windows CI pipeline failure?

Answering to my self :-) , yes but not the feature it self, the tests. fyi this run from https://prow.k8s.io/pr-history/?org=kubernetes&repo=kubernetes&pr=128880 passes without the revert https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/128880/pull-kubernetes-e2e-capz-windows-master/1859388054804893696 . Retriggered to double confirm this. Seems the kubetest2 failure is the last of the three failures where we still have problems.

See https://github.com/kubernetes/kubernetes/pull/128880/files#r1850861491 and the PR for more information.

The CI failure seems to be caused by the e2e pod cleanup logic, but not the feature. ┓( ´∀` )┏

One more update, beside the clean up , after InPlacePodVerticalScaling seems DeleteSync used timeout on some tests was not sufficient

kubernetes/test/e2e_node/memory_manager_test.go

Line 558 in 776fb24

e2epod.NewPodClient(f).DeleteSync(ctx, testPod2.Name, metav1.DeleteOptions{}, 2*time.Minute)

above timeout resulted in failures in kubetest2 pipeline. Same commit shared by pacoxy, tries to test this theory, to see behaviour using e2epod.DefaultPodDeletionTimeout ( which is 3 minutes ) instead.

@esotsal
Copy link

esotsal commented Nov 21, 2024 •
edited
Loading

@pacoxu unfortunately PR landed E_TOO_LATE_IN_THE_CYCLE for us to dig in like this.. so let's do this one right in 1.33.

I agree it is unfortunate that it hasn't made it, to me seems issues were not rooted because of InPlacePodVerticalScaling to be honest. I wish had looked testgrid earlier, but seems we are close fixing those.

@dims
Copy link
Member

dims commented Nov 21, 2024

Agree @esotsal. Good news is that there isn't anything other breakage hiding behind this .. the bleeding has stopped

https://storage.googleapis.com/k8s-triage/index.html?job=.*Serial.*

image

@tallclair
Copy link
Member

Thanks all for holding the high quality bar. I'm disappointed that InPlacePodVerticalScaling won't make it in the v1.32 release, but I'd much rather a smooth rollout!

@tallclair tallclair mentioned this pull request Nov 21, 2024
Merged
@esotsal esotsal mentioned this pull request Dec 4, 2024
Open
@dims dims mentioned this pull request Dec 5, 2024
Merged
@AnishShah AnishShah mentioned this pull request Dec 9, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@aojea aojea Awaiting requested review from aojea

@mrunalp mrunalp Awaiting requested review from mrunalp

@dchen1107 dchen1107 Awaiting requested review from dchen1107

@derekwaynecarr derekwaynecarr Awaiting requested review from derekwaynecarr

@SergeyKanzhelev SergeyKanzhelev Awaiting requested review from SergeyKanzhelev

@tallclair tallclair Awaiting requested review from tallclair

@knabben knabben Awaiting requested review from knabben

Assignees

@rphillips rphillips

@knabben knabben

@aojea aojea

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/kubelet area/test cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. lgtm "Looks good to me", indicates that a PR is ready to be merged. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. release-blocker release-note-none Denotes a PR that doesn't merit a release note. sig/apps Categorizes an issue or PR as relevant to SIG Apps. sig/node Categorizes an issue or PR as relevant to SIG Node. sig/testing Categorizes an issue or PR as relevant to SIG Testing. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
SIG Apps
Archived in project
SIG Node CI/Test Board
Status: Done
SIG Node: In Place Pod Vertical Scaling
Status: Done
Milestone
v1.32
10 participants
@pacoxu @knabben @k8s-ci-robot @dims @rphillips @liggitt @cpanato @aojea @esotsal @tallclair