Add density (batch pods creation latency and resource) and resource performance tests to `test-e2e-node' #29764

coufon · 2016-07-29T01:29:41Z

This PR contains two new tests (migrate from e2e test):

Density test: verify startup latency and resource usage when create a batch of pod with throughput control. Throughput control is done by sleep for an interval between firing concurrently create pod operations.
It tests both batch creation and sequential (back-to-back) creation and report the throughputs.
Verify resource usage of steady state kubelet.

The test creates a new resource controller for `test-node-e2e' (resource_controller.go) which monitors resource through a standalone Cadvisor pod (port 8090) with 1s housekeeping interval.

coufon · 2016-07-29T01:30:27Z

@Random-Liu @yujuhong @dchen1107

Random-Liu · 2016-07-29T01:30:45Z

/cc @kubernetes/sig-node

yujuhong · 2016-07-29T01:39:22Z

@coufon, there are many commits in your PR. Could you squash them and only leave functionally meaningful commits? Thanks!

timothysc · 2016-07-29T16:51:44Z

/cc @rrati @mffiedler @jeremyeder - add to node-vertical.

dchen1107 · 2016-07-29T18:09:54Z

test/e2e_node/e2e_service.go

@@ -251,6 +251,9 @@ func (es *e2eService) startKubeletServer() (*killCmd, error) {
 		"--file-check-frequency", "10s", // Check file frequently so tests won't wait too long
 		"--v", LOG_VERBOSITY_LEVEL, "--logtostderr",
 		"--pod-cidr=10.180.0.0/24", // Assign a fixed CIDR to the node because there is no node controller.
+		"--cgroup-root=/",


This doesn't work with systemd node.

cc/ @yifan-gu on CoreOS configuration, and cc/ @derekwaynecarr on the rest based on "standardized" systemd node.

This flag (and the other cgroup related flags) should not be explicitly set. Just take default kubelet behaviors.

dchen1107 · 2016-07-29T18:11:38Z

Looks good to me overall. We need to figure out how to pass those different cgroup configurations though.

yujuhong · 2016-07-29T18:20:41Z

test/e2e_node/density_test.go

+Copyright 2016 The Kubernetes Authors.
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at


The boilerplate is wrong. If you compare this with other files in the repository, you'll notice that an empty line missing before an after the URL. They probably got lost during copy/paste. Need to fix that to pass the tests.

yujuhong · 2016-07-29T19:27:50Z

test/e2e_node/density_test.go

+		densityTests := []DensityTest{
+			{
+				podsNr:   10,
+				interval: 0 * time.Millisecond,


I'd prefer defining the pod startup throughput (pods per second) and convert it into an interval. Pod throughput is more readable.

The problem is that the `throughput' in configuration may be different from the throughput measured from result. For example, if the interval is zero, the throughput perhaps is infinite. In this case how to define it?

If it was a naming issue, just name it more precisely. E.g., targetThroughput

I am not sure we should have a 0 interval anyway...

dchen1107 · 2016-08-02T16:16:11Z

The build is failed with the error:

# k8s.io/kubernetes/test/e2e_node
test/e2e_node/resource_controller.go:372: undefined: "k8s.io/kubernetes/pkg/util".NewUUID
!!! Error in /go/src/k8s.io/kubernetes/hack/lib/golang.sh:542

Forgot uploading some changes?

…cker dynamically

dchen1107 · 2016-08-03T00:24:33Z

LGTM

k8s-bot · 2016-08-03T11:22:03Z

GCE e2e build/test passed for commit 04f83c7.

k8s-github-robot · 2016-08-03T11:43:14Z

Automatic merge from submit-queue

gmarek · 2016-08-03T13:01:08Z

@lavalamp @fejta - it seems that tests weren't run even though retest-not-required label is not present (or I'm missing something)

gmarek · 2016-08-03T13:13:37Z

I believe it broke k8s build. Reverting - I'll re-revert if it won't help. What's strange is that the build works locally for me... @fejta - it may be some problem with our build system

gmarek · 2016-08-03T15:38:38Z

Revert fixed the build. @coufon @dchen1107 PTAL.

lavalamp · 2016-08-03T19:53:39Z

@apelisse or @foxish can one of you investigate what happened here with the queue?

foxish · 2016-08-03T20:52:00Z

04:43:13.000
I0803 11:43:13.600169 1 submit-queue.go:1149] Skipping retest since head and base sha match previous attempt!
04:43:14.000
I0803 11:43:14.087789 1 github.go:1425] Merging PR# 29764

Looks like it skipped re-test because tests already succeeded at that SHA.

ixdy · 2016-08-03T21:05:35Z

I think the issue is that this PR only builds on linux. e.g. https://github.com/opencontainers/runc/blob/519529febe2e0acb14e39cf54112c292ccb2eabe/libcontainer/cgroups/utils.go, which defines ParseCgroupFile, tags +build linux.

To make SQ faster, we only build for Linux in PR Jenkins. So it passed there.

@coufon

Automatic merge from submit-queue Resolve docker-daemon cgroup issue for both systemd and non-systemd node for node e2e tests Fixed #29827 cc/ @coufon this should unblock your pr: #29764 I validated both containervm image and coreos image, and works as expected. This is also required for adding gci image to node e2e test infrastructure.

Automatic merge from submit-queue Add density (batch pods creation latency and resource) and resource performance tests to `test-e2e-node' built for Linux only This PR adds `+build linux' to density_test.go, resource_usage.go and resource_collector.go to last PR #29764. #29764 fails build because it depends on cgroup which can not be built for os other than Linux.

googlebot added the cla: yes label Jul 29, 2016

yujuhong added sig/node Categorizes an issue or PR as relevant to SIG Node. area/test labels Jul 29, 2016

k8s-github-robot assigned timstclair Jul 29, 2016

k8s-github-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. release-note-label-needed labels Jul 29, 2016

Random-Liu added release-note Denotes a PR that will be considered when it comes time to generate release notes. area/node-e2e and removed release-note-label-needed labels Jul 29, 2016

Random-Liu assigned dchen1107 and yujuhong and unassigned timstclair Jul 29, 2016

dchen1107 unassigned yujuhong Jul 29, 2016

dchen1107 reviewed Jul 29, 2016
View reviewed changes

yujuhong reviewed Jul 29, 2016
View reviewed changes

This was referenced Aug 1, 2016

Docker container name in kubelet is wrong when set container name flag (e.g. --runtime-cgroups=/docker-daemon) for CoreOS image #29827

Closed

Resolve docker-daemon cgroup issue for both systemd and non-systemd node for node e2e tests #29890

Merged

Zhou Fang added 4 commits August 2, 2016 09:20

Add density and resource performance test to test-node-e2e

df55e8f

Update Godeps to include cadvisor client v2

32e1db1

modify resource_collector.go to get container names of kubelet and do…

f3f3e96

…cker dynamically

change the package of uuid in resource_controller

04f83c7

dchen1107 added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 3, 2016

Random-Liu mentioned this pull request Aug 3, 2016

Node E2E: Add serial jenkins job. #29818

Merged

k8s-github-robot merged commit 371c396 into kubernetes:master Aug 3, 2016

gmarek mentioned this pull request Aug 3, 2016

Revert "Add density (batch pods creation latency and resource) and resource performance tests to `test-e2e-node'" #29991

Merged

coufon mentioned this pull request Aug 3, 2016

Add density (batch pods creation latency and resource) and resource performance tests to `test-e2e-node' built for Linux only #30026

Merged

dchen1107 mentioned this pull request Aug 26, 2016

Node Performance Benchmark kubernetes/enhancements#83

Closed

12 tasks

spiffxp mentioned this pull request Aug 16, 2019

The serial regular resource usage tracking e2e tests are too slow #81490

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add density (batch pods creation latency and resource) and resource performance tests to `test-e2e-node' #29764

Add density (batch pods creation latency and resource) and resource performance tests to `test-e2e-node' #29764

coufon commented Jul 29, 2016 •

edited by yujuhong

Loading

coufon commented Jul 29, 2016

Random-Liu commented Jul 29, 2016

yujuhong commented Jul 29, 2016

timothysc commented Jul 29, 2016

dchen1107 Jul 29, 2016

derekwaynecarr Jul 29, 2016

dchen1107 commented Jul 29, 2016

yujuhong Jul 29, 2016

coufon Jul 29, 2016

yujuhong Jul 29, 2016

coufon Jul 29, 2016 •

edited

Loading

yujuhong Aug 1, 2016

dchen1107 commented Aug 2, 2016 •

edited

Loading

dchen1107 commented Aug 3, 2016

k8s-bot commented Aug 3, 2016

k8s-github-robot commented Aug 3, 2016

gmarek commented Aug 3, 2016

gmarek commented Aug 3, 2016

gmarek commented Aug 3, 2016

lavalamp commented Aug 3, 2016

foxish commented Aug 3, 2016

ixdy commented Aug 3, 2016

Add density (batch pods creation latency and resource) and resource performance tests to `test-e2e-node' #29764

Add density (batch pods creation latency and resource) and resource performance tests to `test-e2e-node' #29764

Conversation

coufon commented Jul 29, 2016 • edited by yujuhong Loading

coufon commented Jul 29, 2016

Random-Liu commented Jul 29, 2016

yujuhong commented Jul 29, 2016

timothysc commented Jul 29, 2016

dchen1107 Jul 29, 2016

Choose a reason for hiding this comment

derekwaynecarr Jul 29, 2016

Choose a reason for hiding this comment

dchen1107 commented Jul 29, 2016

yujuhong Jul 29, 2016

Choose a reason for hiding this comment

coufon Jul 29, 2016

Choose a reason for hiding this comment

yujuhong Jul 29, 2016

Choose a reason for hiding this comment

coufon Jul 29, 2016 • edited Loading

Choose a reason for hiding this comment

yujuhong Aug 1, 2016

Choose a reason for hiding this comment

dchen1107 commented Aug 2, 2016 • edited Loading

dchen1107 commented Aug 3, 2016

k8s-bot commented Aug 3, 2016

k8s-github-robot commented Aug 3, 2016

gmarek commented Aug 3, 2016

gmarek commented Aug 3, 2016

gmarek commented Aug 3, 2016

lavalamp commented Aug 3, 2016

foxish commented Aug 3, 2016

ixdy commented Aug 3, 2016

coufon commented Jul 29, 2016 •

edited by yujuhong

Loading

coufon Jul 29, 2016 •

edited

Loading

dchen1107 commented Aug 2, 2016 •

edited

Loading