Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Customize cAdvisor event storage policy #24698

Closed
timstclair opened this issue Apr 23, 2016 · 15 comments
Closed

Customize cAdvisor event storage policy #24698

timstclair opened this issue Apr 23, 2016 · 15 comments
Assignees
Labels
area/cadvisor priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/node Categorizes an issue or PR as relevant to SIG Node.
Milestone

Comments

@timstclair
Copy link

By default cAdvisor stores up to 100,000 events for up to 24 hours. At capacity, that's around O(10MB) that is unused by the Kubelet.

The default values are customized by the --event_storage_{age,event}_limit flags. We should add our own defaults to disable the event store.

/cc @kubernetes/sig-node

@timstclair timstclair self-assigned this Apr 23, 2016
@timstclair timstclair added the sig/node Categorizes an issue or PR as relevant to SIG Node. label Apr 23, 2016
@timstclair timstclair added this to the v1.3 milestone Apr 23, 2016
@vishh
Copy link
Contributor

vishh commented Apr 25, 2016

+1. Does kubelet use these events?

@timstclair
Copy link
Author

Kubelet watches the event stream for Oom events, but never queries the storage. I'd like to just disable event storage completely for the Kubelet's cAdvisor, but that would be a regression if anyone was querying it through the cAdvisor endopoints. It is configurable from a flag though.

@vishh
Copy link
Contributor

vishh commented Apr 25, 2016

If a user wants that feature, they can always run cAdvisor in a DaemonSet :P

On Mon, Apr 25, 2016 at 4:18 PM, Tim St. Clair notifications@github.com
wrote:

Kubelet watches the event stream for Oom events, but never queries the
storage. I'd like to just disable event storage completely for the
Kubelet's cAdvisor, but that would be a regression if anyone was querying
it through the cAdvisor endopoints. It is configurable from a flag though.


You are receiving this because you are on a team that was mentioned.
Reply to this email directly or view it on GitHub
#24698 (comment)

@dchen1107 dchen1107 added area/cadvisor priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. labels Apr 25, 2016
@yujuhong
Copy link
Contributor

Not sure how cadvsior event stream works internally. After kubelet restarts, will it be able to get OOM events that happened in the past N minutes?

@timstclair
Copy link
Author

The event store is an in-memory data structure, so if kubelet restarts it would be lost anyway.

@yujuhong
Copy link
Contributor

I assume cadvisor parses the kernel log again during startup. Is that not correct?

@timstclair
Copy link
Author

Ah, I understand your question. If it restarts, it will reparse the kernel log. The event stream is independent of the event store - so each event will be sent to the watchers and also (independently) added to the event store.

@yujuhong
Copy link
Contributor

yujuhong commented May 3, 2016

I see. Thanks for the clarification!

@roberthbailey
Copy link
Contributor

What is the status of this issue? We are past the feature freeze for 1.3 and this is still part of the milestone.

@timstclair
Copy link
Author

It depends on #25914, which has been LGTM'd for over a week... I'll get this reviewed now though, so it can go in as soon as the dependency is in.

@timstclair
Copy link
Author

Oops, that comment was referring to the PR which addresses this (#24771).

@timothysc
Copy link
Member

Just as a general note, I would still like to off-load events entirely out of the mainline path.

/cc @kubernetes/sig-scalability

@timstclair
Copy link
Author

@timothysc - I don't understand your comment. This issue refers to events generated by cAdvisor from scanning the kernel logs. Were you referring to the same events?

@timothysc
Copy link
Member

@timstclair ohh... I was referring to kubernetes events.

@timstclair
Copy link
Author

Ok, makes sense. This should have no effect on kubernetes events.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/cadvisor priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/node Categorizes an issue or PR as relevant to SIG Node.
Projects
None yet
Development

No branches or pull requests

6 participants