-
Notifications
You must be signed in to change notification settings - Fork 40k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Change default value of deleting-pods-burst to 1 #27422
Conversation
GCE e2e build/test passed for commit aec5dfb. |
LGTM Thanks! |
Automatic merge from submit-queue |
This PR probably broke gce-serial and gce-serail suites. Both fail with:
Reverting it. |
I'm investigating. I found a problem with graceful deletion of Pods on a Kubelet side (run 1500):
I.e. deletion timestamp is updated in every "step". Those logs seem correlated with DELETE calls from namespacecontroller:
I have no idea how default value bump might cause something like this. I suspect another PR that was merged in that time: #26801 @saad-ali @caesarxuchao @yujuhong |
I still don't know why this change might have this effect, but I found a problem with graceful deletion code (I'll send a fix in a sec), that should solve this particular problem. It's kind of strange that it didn't bit us earlier. The problem itself appeared now because for some reason pods we scheduled didn't finish when received SIGTERM from Kubelet. It earlier runs all daemons were deleted quickly, so there were no SIGKILLs required. Now pods refuse to die (which I think is regression - possibly caused by #26801) - @saad-ali to verify - and the bug in graceful deletion logic prevented Kubelet from issueing SIGKILL. |
We should revert that PR then to unblock submit-queue. |
Automatic merge from submit-queue Fix logic of consecutive DELETE calls when gracefull deletion is enabled Previously it was possible to extend delete timestamp to infinity by issuing repeated DELETE calls. This PR prevents that. Ref. discussion in #27422 cc @wojtek-t @smarterclayton @lavalamp @piosz @dchen1107
Fix. #27413