Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Deployment controller should fail and retry gracefully #30833

Closed
janetkuo opened this issue Aug 18, 2016 · 7 comments
Closed

Deployment controller should fail and retry gracefully #30833

janetkuo opened this issue Aug 18, 2016 · 7 comments
Assignees
@janetkuo
Labels
area/workload-api/deployment priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now.

Comments

@janetkuo
Copy link
Member

In rare cases, deployment controller may enter hot fail loop (update --> fail --> retry). We should fail more gracefully so that it doesn't, say, overly spam api server and cause it to become unavailable.

cc @kubernetes/deployment @pwittrock
@adohe-zz
Copy link

@janetkuo is this the same thing as #30028?

@janetkuo
Copy link
Member Author

Similar, but this one is more general (not specific to overlapping deployment), and more about doing things like exponential backoff to prevent hot-fail-loop.

@pwittrock pwittrock added the priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. label Aug 18, 2016
@pwittrock
Copy link
Member

@erictune FYI

@janetkuo
Copy link
Member Author

Found #28684 which added rate limiter to the queue

@janetkuo
Copy link
Member Author

This bug is discovered in an earlier release, and should be fixed on HEAD now. Closing. 🎉

@erictune
Copy link
Member

Does this help at all with #30629

@janetkuo
Copy link
Member Author

#30629 helps this -- we now use rate limiting queue (implements workqueue.RateLimitingInterface) in deployment controller

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@janetkuo janetkuo

Labels
area/workload-api/deployment priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@adohe-zz @janetkuo @pwittrock @erictune