Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ignore failed pods to not block rolling update daemonset #78170

Merged
merged 1 commit into from
Jul 24, 2019
Merged

ignore failed pods to not block rolling update daemonset #78170

merged 1 commit into from
Jul 24, 2019

Conversation

DaiHao
Copy link
Contributor

@DaiHao DaiHao commented May 21, 2019 •
edited
Loading

What type of PR is this?

/kind bug

What this PR does / why we need it:

daemon set rolling update hang when there exist failed pods on node
As long as there exists one failed pod of daemonset, it will block the entire upgrade process.

Which issue(s) this PR fixes:

Fixes #63465

Special notes for your reviewer:

#65309 has used backoff algorithm to avoid hot loops fighting with kubelet on pod recreation, so it's safe to return nil in 'manage' method when there exist failed pod.
If not, 'manage' method returns error which triggers syncDaemonSet method returns directly. After failed pod be deleted, a new pod will be created but may still in failed state if kubelet reject it. So we got a another failed pod and fall into an infinite loop.

/assign @janetkuo @k82cn
/sig apps
/kind bug
/priority important-soon

Does this PR introduce a user-facing change?:

Fix a bug that causes DaemonSet rolling update hang when there exist failed pods.

@k8s-ci-robot k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. release-note Denotes a PR that will be considered when it comes time to generate release notes. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-kind Indicates a PR lacks a `kind/foo` label and requires one. needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels May 21, 2019
@k8s-ci-robot
Copy link
Contributor

Hi @DaiHao. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. sig/apps Categorizes an issue or PR as relevant to SIG Apps. kind/bug Categorizes issue or PR as related to a bug. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-kind Indicates a PR lacks a `kind/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels May 21, 2019
@DaiHao DaiHao changed the title ignore failed pods to not stuck rolling update daemonset ignore failed pods to not block rolling update daemonset May 21, 2019
@k82cn
Copy link
Member

k82cn commented May 21, 2019

/assign

I'd like to go through this firstly to see what happened there :)

@k8s-ci-robot
Copy link
Contributor

@DaiHao: You must be a member of the kubernetes/kubernetes-milestone-maintainers GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility.

In response to this:

/milestone v1.15

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@DaiHao
Copy link
Contributor Author

DaiHao commented May 27, 2019

/assign

I'd like to go through this firstly to see what happened there :)

ptal， it's supposed to be merged in v1.15.

@zhangxiaoyu-zidif
Copy link
Contributor

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels May 28, 2019
@DaiHao
Copy link
Contributor Author

DaiHao commented May 28, 2019

/test pull-kubernetes-kubemark-e2e-gce-big

@DaiHao
Copy link
Contributor Author

DaiHao commented Jun 14, 2019

@k82cn @janetkuo this pr has pending for a while, ptal

@DaiHao
Copy link
Contributor Author

DaiHao commented Jul 11, 2019

ping @k82cn

@ScorpioCPH
Copy link
Contributor

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 12, 2019
@janetkuo
Copy link
Member

/approve

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: DaiHao, janetkuo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 24, 2019
@DaiHao
Copy link
Contributor Author

DaiHao commented Jul 24, 2019

/test pull-kubernetes-kubemark-e2e-gce-big

@k8s-ci-robot k8s-ci-robot merged commit bbdd055 into kubernetes:master Jul 24, 2019
@evertrain evertrain mentioned this pull request Jul 19, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@janetkuo janetkuo janetkuo approved these changes

@mortent mortent Awaiting requested review from mortent

@smarterclayton smarterclayton Awaiting requested review from smarterclayton

Assignees

@janetkuo janetkuo

@ScorpioCPH ScorpioCPH

@k82cn k82cn

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/apps Categorizes an issue or PR as relevant to SIG Apps. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

daemonset rollingupdate hang
6 participants
@DaiHao @k8s-ci-robot @k82cn @zhangxiaoyu-zidif @ScorpioCPH @janetkuo