Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Umbrella bug: Too many lists in k8s cluster #6564

Closed
2 of 3 tasks
fgrzadkowski opened this issue Apr 8, 2015 · 10 comments
Closed
2 of 3 tasks

Umbrella bug: Too many lists in k8s cluster #6564

fgrzadkowski opened this issue Apr 8, 2015 · 10 comments
Assignees
Labels
priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/scalability Categorizes an issue or PR as relevant to SIG Scalability.
Milestone

Comments

@fgrzadkowski
Copy link
Contributor

As reported in #6514 lists can be very heavy. Currently majority of CPU usage in apiserver is caused by list handler. We should limit this significantly.

Subtasks:

@wojtek-t @lavalamp @davidopp

@fgrzadkowski fgrzadkowski added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/scalability Categorizes an issue or PR as relevant to SIG Scalability. team/master labels Apr 8, 2015
@fgrzadkowski fgrzadkowski self-assigned this Apr 8, 2015
@fgrzadkowski fgrzadkowski added this to the v1.0 milestone Apr 8, 2015
@wojtek-t
Copy link
Member

wojtek-t commented Apr 8, 2015

#6513 may also be part of this effort.

@lavalamp
Copy link
Member

lavalamp commented Apr 8, 2015

Please see: #4817 (comment)

@bprashanth
Copy link
Contributor

@fgrzadkowski I don't completely understand the tradeoffs involved in swapping out those lists for watches, thoughts on how we can avoid an Umbrella bug with %s/lists/watches down the line?

@fgrzadkowski
Copy link
Contributor Author

@bprashanth Sorry, but I don't understand your comment. I don't think there are any tradeoffs here. Semantically we will have exactly the same logic, but without reading the same objects multiple times.

@erictune
Copy link
Member

Listing is less error-prone than watching. We should definitely support
clients that want to list frequently. The reported CPU saturation does not
make sense given the quantities of data involves, suggesting a pathological
bug. That should be fixed first.

On Tue, Apr 14, 2015 at 12:00 AM, Filip Grzadkowski <
notifications@github.com> wrote:

@bprashanth https://github.com/bprashanth Sorry, but I don't understand
your comment. I don't think there are any tradeoffs here. Semantically we
will have exactly the same logic, but without reading the same objects
multiple times.


Reply to this email directly or view it on GitHub
#6564 (comment)
.

@bprashanth
Copy link
Contributor

@fgrzadkowski watching involves continuously filtering every change on the server as it occurs, while listing is just filtering all objects as they currently exist (so if an object goes from rv 1->100, all 100 are sent down the watch, but only the 100th will be sent down the list). I would expect watches to add some constant strain, especially since the watch also does a periodic list every 5 minutes.

@fgrzadkowski
Copy link
Contributor Author

@erictune Please see my example in this #6514 (comment)

@bprashanth I agree that listing is easier for some clients, but:
a) we already have simple libraries that make using watch very easy (watch-based cache)
b) current implementation of label/field selectors is sub-optimal
c) if we control some clients (such as controllers) we should make them as light-weight (from apiserver perspective) as possible. Obviously we will have to support lists and make them faster (see #6514 and #6800) but that's a completely separate story

@fgrzadkowski
Copy link
Contributor Author

After fixing #6518 situation improved but we still have high number (in terms of CPU usage) of lists. This happens in density test when replication controller creates new pods. I'm debugging what's the source (I don't think it's RC).

@wojtek-t

list_profile

@timothysc
Copy link
Member

@rrati ^

@davidopp
Copy link
Member

Only remaining issue is #6562 so closing this one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/scalability Categorizes an issue or PR as relevant to SIG Scalability.
Projects
None yet
Development

No branches or pull requests

7 participants