Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Soak test based on serve_hostname image #5750

Merged
merged 1 commit into from
Mar 23, 2015
Merged

Soak test based on serve_hostname image #5750

merged 1 commit into from
Mar 23, 2015

Conversation

satnam6502
Copy link
Contributor

This is a simple soak test that performs the following actions when executed for the GCE provider:

  • Obtains information about what cluster to use for the test from $HOME/.kube/.kubeconfig
  • Examines how many nodes are on the cluster (say N).
  • Creates a fresh namespace for the test using a random suffix.
  • Instantiates M (default 1) serve_hostname pods on each node.
  • Creates a service that captures the pods serve-hostname-N-M.
  • Repeatedly issues Q (default 1,000) queries to the service.
  • Reports the distribution of how many queries were handled by each pod (we expect and even distribution).
  • It does this in a loop forever.

Keeping this running constitutes a simple soak test.

$ ./serve_hostnames --per_node=5
Nodes found on this cluster:
0: kubernetes-minion-8ezp.c.kubernetes-satnam.internal
1: kubernetes-minion-fga5.c.kubernetes-satnam.internal
2: kubernetes-minion-p2u9.c.kubernetes-satnam.internal
3: kubernetes-minion-viip.c.kubernetes-satnam.internal
Using namespace serve-hostnames-8826 for this test.
Creating pod serve-hostnames-8826/serve-hostname-0-0 on node kubernetes-minion-8ezp.c.kubernetes-satnam.internal ... [done]
Creating pod serve-hostnames-8826/serve-hostname-0-1 on node kubernetes-minion-8ezp.c.kubernetes-satnam.internal ... [done]
Creating pod serve-hostnames-8826/serve-hostname-0-2 on node kubernetes-minion-8ezp.c.kubernetes-satnam.internal ... [done]
Creating pod serve-hostnames-8826/serve-hostname-0-3 on node kubernetes-minion-8ezp.c.kubernetes-satnam.internal ... [done]
Creating pod serve-hostnames-8826/serve-hostname-0-4 on node kubernetes-minion-8ezp.c.kubernetes-satnam.internal ... [done]
Creating pod serve-hostnames-8826/serve-hostname-1-0 on node kubernetes-minion-fga5.c.kubernetes-satnam.internal ... [done]
Creating pod serve-hostnames-8826/serve-hostname-1-1 on node kubernetes-minion-fga5.c.kubernetes-satnam.internal ... [done]
Creating pod serve-hostnames-8826/serve-hostname-1-2 on node kubernetes-minion-fga5.c.kubernetes-satnam.internal ... [done]
Creating pod serve-hostnames-8826/serve-hostname-1-3 on node kubernetes-minion-fga5.c.kubernetes-satnam.internal ... [done]
Creating pod serve-hostnames-8826/serve-hostname-1-4 on node kubernetes-minion-fga5.c.kubernetes-satnam.internal ... [done]
Creating pod serve-hostnames-8826/serve-hostname-2-0 on node kubernetes-minion-p2u9.c.kubernetes-satnam.internal ... [done]
Creating pod serve-hostnames-8826/serve-hostname-2-1 on node kubernetes-minion-p2u9.c.kubernetes-satnam.internal ... [done]
Creating pod serve-hostnames-8826/serve-hostname-2-2 on node kubernetes-minion-p2u9.c.kubernetes-satnam.internal ... [done]
Creating pod serve-hostnames-8826/serve-hostname-2-3 on node kubernetes-minion-p2u9.c.kubernetes-satnam.internal ... [done]
Creating pod serve-hostnames-8826/serve-hostname-2-4 on node kubernetes-minion-p2u9.c.kubernetes-satnam.internal ... [done]
Creating pod serve-hostnames-8826/serve-hostname-3-0 on node kubernetes-minion-viip.c.kubernetes-satnam.internal ... [done]
Creating pod serve-hostnames-8826/serve-hostname-3-1 on node kubernetes-minion-viip.c.kubernetes-satnam.internal ... [done]
Creating pod serve-hostnames-8826/serve-hostname-3-2 on node kubernetes-minion-viip.c.kubernetes-satnam.internal ... [done]
Creating pod serve-hostnames-8826/serve-hostname-3-3 on node kubernetes-minion-viip.c.kubernetes-satnam.internal ... [done]
Creating pod serve-hostnames-8826/serve-hostname-3-4 on node kubernetes-minion-viip.c.kubernetes-satnam.internal ... [done]
Creating service serve-hostnames ... [done]
Wait for the serve-hostname pods to be ready
serve-hostnames-8826/serve-hostname-0-0 is running
serve-hostnames-8826/serve-hostname-0-1 is running
serve-hostnames-8826/serve-hostname-0-2 is running
serve-hostnames-8826/serve-hostname-0-3 is running
serve-hostnames-8826/serve-hostname-0-4 is running
serve-hostnames-8826/serve-hostname-1-0 is running
serve-hostnames-8826/serve-hostname-1-1 is running
serve-hostnames-8826/serve-hostname-1-2 is running
serve-hostnames-8826/serve-hostname-1-3 is running
serve-hostnames-8826/serve-hostname-1-4 is running
serve-hostnames-8826/serve-hostname-2-0 is running
serve-hostnames-8826/serve-hostname-2-1 is running
serve-hostnames-8826/serve-hostname-2-2 is running
serve-hostnames-8826/serve-hostname-2-3 is running
serve-hostnames-8826/serve-hostname-2-4 is running
serve-hostnames-8826/serve-hostname-3-0 is running
serve-hostnames-8826/serve-hostname-3-1 is running
serve-hostnames-8826/serve-hostname-3-2 is running
serve-hostnames-8826/serve-hostname-3-3 is running
serve-hostnames-8826/serve-hostname-3-4 is running
Waiting for the endpoints to propagate... finished waiting
serve-hostname-1-2: 61  serve-hostname-0-3: 51  serve-hostname-2-1: 46  serve-hostname-2-0: 42  serve-hostname-1-1: 44  serve-hostname-0-1: 42  serve-hostname-0-0: 62  serve-hostname-1-0: 49  serve-hostname-1-4: 39  serve-hostname-2-3: 64  serve-hostname-3-3: 40  serve-hostname-2-2: 56  serve-hostname-1-3: 43  serve-hostname-0-4: 58  serve-hostname-0-2: 46  serve-hostname-2-4: 57  serve-hostname-3-1: 59  serve-hostname-3-2: 51  serve-hostname-3-4: 42  serve-hostname-3-0: 48  
Took 51.191636354s
serve-hostname-3-1: 55  serve-hostname-1-3: 50  serve-hostname-1-4: 52  serve-hostname-1-0: 49  serve-hostname-2-1: 41  serve-hostname-0-1: 42  serve-hostname-2-2: 46  serve-hostname-0-4: 47  serve-hostname-0-3: 38  serve-hostname-2-3: 38  serve-hostname-0-0: 58  serve-hostname-1-2: 63  serve-hostname-3-3: 59  serve-hostname-2-4: 65  serve-hostname-3-0: 51  serve-hostname-1-1: 49  serve-hostname-3-2: 48  serve-hostname-2-0: 48  serve-hostname-0-2: 48  serve-hostname-3-4: 53  
Took 52.022849323s
serve-hostname-3-2: 58  serve-hostname-1-2: 48  serve-hostname-1-1: 48  serve-hostname-2-1: 53  serve-hostname-1-0: 46  serve-hostname-1-3: 52  serve-hostname-3-1: 45  serve-hostname-2-4: 43  serve-hostname-2-3: 48  serve-hostname-2-0: 70  serve-hostname-3-4: 47  serve-hostname-0-1: 51  serve-hostname-0-3: 64  serve-hostname-0-0: 50  serve-hostname-0-4: 41  serve-hostname-2-2: 44  serve-hostname-3-3: 43  serve-hostname-3-0: 54  serve-hostname-0-2: 54  serve-hostname-1-4: 41  
Took 51.382949407s
serve-hostname-0-0: 45  serve-hostname-0-3: 51  serve-hostname-3-2: 40  serve-hostname-1-3: 61  serve-hostname-2-1: 47  serve-hostname-1-2: 49  serve-hostname-3-0: 57  serve-hostname-2-3: 53  serve-hostname-1-4: 54  serve-hostname-3-1: 58  serve-hostname-2-4: 42  serve-hostname-0-1: 64  serve-hostname-0-2: 40  serve-hostname-1-0: 55  serve-hostname-2-0: 43  serve-hostname-3-3: 50  serve-hostname-0-4: 53  serve-hostname-2-2: 57  serve-hostname-1-1: 44  serve-hostname-3-4: 37  
Took 52.76047085s
serve-hostname-1-0: 45  serve-hostname-3-2: 56  serve-hostname-0-1: 48  serve-hostname-2-3: 55  serve-hostname-0-4: 62  serve-hostname-3-3: 58  serve-hostname-2-0: 59  serve-hostname-0-0: 47  serve-hostname-3-1: 42  serve-hostname-1-1: 43  serve-hostname-2-1: 69  serve-hostname-3-0: 48  serve-hostname-1-2: 53  serve-hostname-1-3: 47  serve-hostname-1-4: 43  serve-hostname-2-4: 48  serve-hostname-2-2: 44  serve-hostname-0-2: 48  serve-hostname-3-4: 41  serve-hostname-0-3: 44  
Took 51.717851856s
serve-hostname-2-2: 46  serve-hostname-1-3: 43  serve-hostname-1-1: 49  serve-hostname-3-4: 55  serve-hostname-0-3: 52  serve-hostname-1-0: 47  serve-hostname-3-1: 45  serve-hostname-0-1: 54  serve-hostname-1-4: 32  serve-hostname-0-2: 64  serve-hostname-2-1: 52  serve-hostname-3-2: 47  serve-hostname-3-3: 51  serve-hostname-3-0: 55  serve-hostname-2-3: 54  serve-hostname-0-0: 42  serve-hostname-1-2: 66  serve-hostname-0-4: 55  serve-hostname-2-0: 46  serve-hostname-2-4: 45  
Took 52.904848788s
serve-hostname-0-0: 46  serve-hostname-2-1: 54  serve-hostname-0-3: 30  serve-hostname-3-4: 52  serve-hostname-2-0: 55  serve-hostname-1-0: 49  serve-hostname-3-0: 55  serve-hostname-0-1: 51  serve-hostname-1-2: 43  serve-hostname-3-1: 56  serve-hostname-1-1: 48  serve-hostname-0-2: 51  serve-hostname-3-2: 46  serve-hostname-1-3: 52  serve-hostname-1-4: 46  serve-hostname-2-3: 62  serve-hostname-3-3: 48  serve-hostname-2-2: 56  serve-hostname-2-4: 49  serve-hostname-0-4: 51  
Took 52.712764611s
serve-hostname-3-4: 61  serve-hostname-0-4: 52  serve-hostname-0-3: 44  serve-hostname-1-0: 60  serve-hostname-1-1: 42  serve-hostname-2-2: 45  serve-hostname-1-3: 55  serve-hostname-3-2: 56  serve-hostname-3-3: 59  serve-hostname-3-1: 53  serve-hostname-1-4: 62  serve-hostname-0-2: 40  serve-hostname-0-0: 45  serve-hostname-1-2: 46  serve-hostname-2-1: 36  serve-hostname-0-1: 44  serve-hostname-3-0: 62  serve-hostname-2-4: 45  serve-hostname-2-0: 54  serve-hostname-2-3: 39  
Took 53.611146825s
serve-hostname-1-4: 55  serve-hostname-0-0: 37  serve-hostname-3-2: 51  serve-hostname-3-0: 58  serve-hostname-1-2: 41  serve-hostname-2-3: 47  serve-hostname-1-1: 53  serve-hostname-1-3: 50  serve-hostname-0-2: 52  serve-hostname-1-0: 49  serve-hostname-0-1: 52  serve-hostname-0-3: 53  serve-hostname-2-0: 43  serve-hostname-3-3: 48  serve-hostname-3-1: 58  serve-hostname-3-4: 49  serve-hostname-0-4: 60  serve-hostname-2-4: 47  serve-hostname-2-2: 54  serve-hostname-2-1: 43  
Took 52.354442773s

@satnam6502
Copy link
Contributor Author

@roberthbailey @jayunit100 @davidopp @lavalamp @thockin @jlowdermilk
Related issue #3137

@roberthbailey roberthbailey self-assigned this Mar 21, 2015
roberthbailey
roberthbailey reviewed Mar 21, 2015
View reviewed changes
test/soak/serve_hostnames/serve_hostnames.go
perNode = flag.Int("per_node", 1, "Number of hostname pods per node")
)

const (
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need for a block with just one constant.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed, thanks.

@satnam6502
Copy link
Contributor Author

PTAL. I think I've dealt with all the excellent suggestions. For now, I have changed some Fatals to Warnings and then I return to make sure the defer cleanup occurs. I report any nodes that did not respond with a Warningf. In a future iteration I will handle SIGINT and SIGKILL to make sure the defer occurs. It is now possible to specify the number of iterations (see below) so this can be added to some suite or regularly run builds for examples etc. to avoid bit-rot. I also now create the service first and then the pods so the 30 second wait for the endpoints to propagate has been removed.

$ ./serve_hostnames --up_to=1
I0321 11:12:06.071849   76721 serve_hostnames.go:53] Starting serve_hostnames soak test with queries=1000 and perNode=1 upTo=1
I0321 11:12:06.481369   76721 serve_hostnames.go:78] Nodes found on this cluster:
I0321 11:12:06.481386   76721 serve_hostnames.go:80] 0: kubernetes-minion-6gen.c.kubernetes-satnam2.internal
I0321 11:12:06.481391   76721 serve_hostnames.go:80] 1: kubernetes-minion-ivc1.c.kubernetes-satnam2.internal
I0321 11:12:06.481395   76721 serve_hostnames.go:80] 2: kubernetes-minion-nkl2.c.kubernetes-satnam2.internal
I0321 11:12:06.481399   76721 serve_hostnames.go:80] 3: kubernetes-minion-qjrh.c.kubernetes-satnam2.internal
I0321 11:12:06.481417   76721 serve_hostnames.go:86] Using namespace serve-hostnames-1091 for this test.
I0321 11:12:06.481423   76721 serve_hostnames.go:89] Creating service serve-hostnames
I0321 11:12:06.569275   76721 serve_hostnames.go:123] Creating pod serve-hostnames-1091/serve-hostname-0-0 on node kubernetes-minion-6gen.c.kubernetes-satnam2.internal
I0321 11:12:06.649259   76721 serve_hostnames.go:123] Creating pod serve-hostnames-1091/serve-hostname-1-0 on node kubernetes-minion-ivc1.c.kubernetes-satnam2.internal
I0321 11:12:06.737740   76721 serve_hostnames.go:123] Creating pod serve-hostnames-1091/serve-hostname-2-0 on node kubernetes-minion-nkl2.c.kubernetes-satnam2.internal
I0321 11:12:06.858153   76721 serve_hostnames.go:123] Creating pod serve-hostnames-1091/serve-hostname-3-0 on node kubernetes-minion-qjrh.c.kubernetes-satnam2.internal
I0321 11:12:06.966499   76721 serve_hostnames.go:158] Waiting for the serve-hostname pods to be ready
I0321 11:12:53.014380   76721 serve_hostnames.go:175] serve-hostnames-1091/serve-hostname-0-0 is running
I0321 11:13:18.486994   76721 serve_hostnames.go:175] serve-hostnames-1091/serve-hostname-1-0 is running
I0321 11:13:18.596576   76721 serve_hostnames.go:175] serve-hostnames-1091/serve-hostname-2-0 is running
I0321 11:13:18.670425   76721 serve_hostnames.go:175] serve-hostnames-1091/serve-hostname-3-0 is running
I0321 11:14:28.024996   76721 serve_hostnames.go:202] serve-hostname-2-0: 265
I0321 11:14:28.025029   76721 serve_hostnames.go:202] serve-hostname-0-0: 261
I0321 11:14:28.025039   76721 serve_hostnames.go:202] serve-hostname-3-0: 264
I0321 11:14:28.025049   76721 serve_hostnames.go:202] serve-hostname-1-0: 210
I0321 11:14:28.025065   76721 serve_hostnames.go:213] Iteration 0 took 1m9.35461398s for 1000 queries
I0321 11:14:28.025091   76721 serve_hostnames.go:150] Cleaning up pods
I0321 11:14:28.349487   76721 serve_hostnames.go:111] Cleaning up service

@satnam6502 satnam6502 force-pushed the soak branch 3 times, most recently from 11115fe to a26b416 Compare March 21, 2015 18:31
@satnam6502 satnam6502 force-pushed the soak branch 3 times, most recently from 3f1d74b to a375733 Compare March 21, 2015 21:39
@satnam6502 satnam6502 mentioned this pull request Mar 21, 2015
Closed
@jayunit100
Copy link
Member

Cool idea ! So, this is a specific kind of soak, i.e. i think this test is specifically testing the amount of times a service is forwarded to one machine, versus another.

So shall we maybe call it svc_soak.go ?

then i think the other tests can be on the lines of net-soak, cpu-soak, all-soak (this i think is what I'm working on atm)...

lgtm and hope to try it out soon.

@satnam6502
Copy link
Contributor Author

I am currently running it on a 50 node cluster with N pods per node. I am also producing rc_serve_hostnames which does not explicitly pin pods to nodes but instead makes use of a replication controller and it also randomly kills pods and adds pods and checks that the overall behaviour of the system is fine. I am calling this test cauldron because it keeps the number of pods bubbling along at roughly the same number.

@satnam6502
Copy link
Contributor Author

Even 10 pods per node is too much.

I0322 05:28:18.910522    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-49-3 is running
I0322 05:28:18.917045    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-49-4 is running
Timeout, server 104.155.236.85 not responding.

@satnam6502 satnam6502 mentioned this pull request Mar 22, 2015
Closed
@satnam6502
Copy link
Contributor Author

5 pods per node also does not work.

@roberthbailey roberthbailey added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 23, 2015
roberthbailey added a commit that referenced this pull request Mar 23, 2015
@roberthbailey
Merge pull request #5750 from satnam6502/soak
b46276e 
Soak test based on serve_hostname image
@roberthbailey roberthbailey merged commit b46276e into kubernetes:master Mar 23, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@roberthbailey roberthbailey

Labels
lgtm "Looks good to me", indicates that a PR is ready to be merged.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@satnam6502 @jayunit100 @roberthbailey @googlebot @davidopp