Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use the summary_api with heapster by default #22542

Merged
merged 1 commit into from
Mar 4, 2016
Merged

Use the summary_api with heapster by default #22542

merged 1 commit into from
Mar 4, 2016
+8 −8

Conversation

timstclair
Copy link

Use the new summary api by default with heapster.

We have conducted correctness and performance testing on the new API, and found it to be sound. There are two changes in the metrics heapster gathers from the new API:

  1. Pod infrastructure container stats are not included on pod MetricSets. CPU usage for the infra container is zero after an initial small burst (~50 milli-core-seconds), and memory RSS is around 1.3 MiB.
  2. Filesystem stats are handled differently. The new API reports rootfs (resource ID = '/') and log (resource ID = 'logs') FS stats on containers, and volume stats (resource ID = volume name) on pods.

Performance of the heapster source with the summary API is improved over the cAdvisor API. A heapster instance with the old source and one with the new source were run side-by-side on the same cluster and measured while running the density 30-pod e2e test. Results were gathered on a real 3-node and 50-node cluster, as well as on a simulated 30, 60, and 300 node cluster. The simulated results were gathered by running multiple heapster sources (metric scraper workers) per node (i.e. metrics were scraped & aggregated from each node multiple times). The 99th percentile usage was:

container                                                                  cpu(cores) memory(MB)
TEST: 30 pods x 3 nodes, 1 source per node, 13m42.809402175s
"heapster-perf-kubelet-j30bg/heapster-test"                                0.006      27.46
"heapster-perf-summary-vcvl6/heapster-test"                                0.005      25.25

TEST: 30 pods x 50 nodes, 1 source per node, 32m22.336210862s
"heapster-perf-kubelet-7jnaj/heapster"                                           0.067      240.07
"heapster-perf-summary-jlylm/heapster"                                           0.045      215.76

TEST: 30 pods x 3 nodes, 10 sources per node, 13m52.850112505s
"heapster-perf-kubelet-ip0os/heapster-test"                                0.019      42.32
"heapster-perf-summary-ihmnv/heapster-test"                                0.008      26.67

TEST: 30 pods x 3 nodes, 20 sources per node, 13m34.660766648s
"heapster-perf-kubelet-vgzz0/heapster-test"                                0.035      51.28
"heapster-perf-summary-3g24j/heapster-test"                                0.011      27.62

TEST: 30 pods x 3 nodes, 100 sources per node, 13m50.085140262s
"heapster-perf-kubelet-1ln3r/heapster-test"                                0.229      152.20
"heapster-perf-summary-57xg4/heapster-test"                                0.047      31.00

Full results here

/cc @dchen1107 @mwielgus @piosz @vishh @pwittrock

@timstclair timstclair added this to the v1.2 milestone Mar 4, 2016
@timstclair
Copy link
Author

/cc @kubernetes/sig-node @kubernetes/sig-scalability

@k8s-github-robot
Copy link

Labelling this PR as size/S

@k8s-github-robot k8s-github-robot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Mar 4, 2016
@k8s-bot
Copy link

k8s-bot commented Mar 4, 2016

GCE e2e build/test passed for commit 1d90394.

@dchen1107 dchen1107 added release-note Denotes a PR that will be considered when it comes time to generate release notes. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. labels Mar 4, 2016
@dchen1107 dchen1107 assigned dchen1107 and unassigned fgrzadkowski Mar 4, 2016
@dchen1107
Copy link
Member

I assigned this one to myself since it is weekend for Warsaw folks already. @fgrzadkowski I want to this one in as soon as possible, so that we can monitor a couple of builds before MTV weekend.

@dchen1107
Copy link
Member

LGTM

@dchen1107 dchen1107 added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 4, 2016
ArtfulCoder added a commit that referenced this pull request Mar 4, 2016
@ArtfulCoder
Merge pull request #22542 from timstclair/heapster
c47564f 
Use the summary_api with heapster by default
@ArtfulCoder ArtfulCoder merged commit c47564f into kubernetes:master Mar 4, 2016
@fgrzadkowski
Copy link
Contributor

LGTM

Thanks!

@dchen1107 dchen1107 mentioned this pull request Mar 23, 2016
Open
@timstclair timstclair deleted the heapster branch August 31, 2016 23:37
@andyxning andyxning mentioned this pull request Sep 8, 2017
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@dchen1107 dchen1107

Labels
lgtm "Looks good to me", indicates that a PR is ready to be merged. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.
Projects
None yet
Milestone
v1.2
Development

Successfully merging this pull request may close these issues.
9 participants
@timstclair @k8s-github-robot @k8s-bot @dchen1107 @fgrzadkowski @ArtfulCoder @googlebot @eparis @tallclair