Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Change defaults, retry on errors and report stats for serve_hostnames #5881

Merged
merged 1 commit into from
Mar 25, 2015
Merged

Change defaults, retry on errors and report stats for serve_hostnames #5881

merged 1 commit into from
Mar 25, 2015
+100 −41

Conversation

satnam6502
Copy link
Contributor

No description provided.

@satnam6502 satnam6502 changed the title Change defaults, retry on errors and report stats Change defaults, retry on errors and report stats for serve_hostnames Mar 24, 2015
@satnam6502
Copy link
Contributor Author

Here is a representative run. Worryingly one of the nodes never fields any requests.

$ ./serve_hostnames 
I0324 16:15:04.218410   10550 serve_hostnames.go:60] Starting serve_hostnames soak test with queries=100 and perNode=1 upTo=1
I0324 16:15:04.549121   10550 serve_hostnames.go:85] Nodes found on this cluster:
I0324 16:15:04.549183   10550 serve_hostnames.go:87] 0: kubernetes-minion-5h4m.c.kubernetes-satnam.internal
I0324 16:15:04.549211   10550 serve_hostnames.go:87] 1: kubernetes-minion-9i4n.c.kubernetes-satnam.internal
I0324 16:15:04.549233   10550 serve_hostnames.go:87] 2: kubernetes-minion-d0yo.c.kubernetes-satnam.internal
I0324 16:15:04.549256   10550 serve_hostnames.go:87] 3: kubernetes-minion-jay1.c.kubernetes-satnam.internal
I0324 16:15:04.549314   10550 serve_hostnames.go:93] Using namespace serve-hostnames-3700 for this test.
I0324 16:15:04.549344   10550 serve_hostnames.go:96] Creating service serve-hostnames-3700/serve-hostnames
I0324 16:15:04.601298   10550 serve_hostnames.go:116] Service create serve-hostnames-3700/server-hostnames took 51.922493ms
I0324 16:15:04.601378   10550 serve_hostnames.go:146] Creating pod serve-hostnames-3700/serve-hostname-0-0 on node kubernetes-minion-5h4m.c.kubernetes-satnam.internal
I0324 16:15:04.655480   10550 serve_hostnames.go:166] Pod create serve-hostnames-3700/serve-hostname-0-0 request took 54.065638ms
I0324 16:15:04.655562   10550 serve_hostnames.go:146] Creating pod serve-hostnames-3700/serve-hostname-1-0 on node kubernetes-minion-9i4n.c.kubernetes-satnam.internal
I0324 16:15:04.707213   10550 serve_hostnames.go:166] Pod create serve-hostnames-3700/serve-hostname-1-0 request took 51.600179ms
I0324 16:15:04.707286   10550 serve_hostnames.go:146] Creating pod serve-hostnames-3700/serve-hostname-2-0 on node kubernetes-minion-d0yo.c.kubernetes-satnam.internal
I0324 16:15:04.756102   10550 serve_hostnames.go:166] Pod create serve-hostnames-3700/serve-hostname-2-0 request took 48.766829ms
I0324 16:15:04.756170   10550 serve_hostnames.go:146] Creating pod serve-hostnames-3700/serve-hostname-3-0 on node kubernetes-minion-jay1.c.kubernetes-satnam.internal
I0324 16:15:04.806265   10550 serve_hostnames.go:166] Pod create serve-hostnames-3700/serve-hostname-3-0 request took 50.059389ms
I0324 16:15:04.806345   10550 serve_hostnames.go:192] Waiting for the serve-hostname pods to be ready
I0324 16:16:35.665703   10550 serve_hostnames.go:209] serve-hostnames-3700/serve-hostname-0-0 is running
I0324 16:16:35.707723   10550 serve_hostnames.go:209] serve-hostnames-3700/serve-hostname-1-0 is running
I0324 16:16:55.923535   10550 serve_hostnames.go:209] serve-hostnames-3700/serve-hostname-2-0 is running
I0324 16:16:55.965506   10550 serve_hostnames.go:209] serve-hostnames-3700/serve-hostname-3-0 is running
I0324 16:16:56.049752   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 41.989524ms
I0324 16:16:56.091592   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 41.76781ms
I0324 16:16:56.131743   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 40.09725ms
I0324 16:16:56.172905   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 41.073594ms
I0324 16:16:56.214507   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 41.514842ms
I0324 16:16:56.255891   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 41.294787ms
I0324 16:16:56.296570   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 40.589434ms
I0324 16:16:56.336862   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 40.217138ms
I0324 16:16:56.376680   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 39.780395ms
I0324 16:16:56.417169   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 40.451971ms
I0324 16:16:56.457995   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 40.79014ms
I0324 16:16:56.498469   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 40.434803ms
I0324 16:16:56.538856   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 40.338595ms
I0324 16:16:56.585603   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 46.664626ms
I0324 16:16:56.626298   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 40.607817ms
I0324 16:16:56.666857   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 40.470281ms
I0324 16:16:56.713475   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 46.532193ms
I0324 16:16:56.754494   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 40.930458ms
I0324 16:16:56.795696   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 41.116433ms
I0324 16:16:56.836784   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 41.006411ms
I0324 16:16:56.877428   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 40.572581ms
I0324 16:16:56.918216   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 40.710982ms
I0324 16:16:56.958953   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 40.650437ms
I0324 16:16:57.001063   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 42.071887ms
I0324 16:16:57.041825   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 40.724642ms
I0324 16:16:57.083487   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 41.623032ms
I0324 16:16:57.124454   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 40.927483ms
I0324 16:16:57.165588   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 41.087221ms
I0324 16:16:57.207069   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 41.409592ms
I0324 16:16:57.247777   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 40.596264ms
I0324 16:16:57.288667   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 40.800618ms
I0324 16:16:57.329956   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 41.195672ms
I0324 16:16:57.371566   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 41.521254ms
I0324 16:16:57.412413   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 40.763588ms
I0324 16:16:57.453154   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 40.701189ms
I0324 16:16:57.494663   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 41.475887ms
I0324 16:16:57.535095   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 40.398952ms
I0324 16:16:57.575415   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 40.286975ms
I0324 16:16:57.627246   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 51.784909ms
I0324 16:16:57.667939   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 40.627148ms
I0324 16:16:57.710138   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 42.141809ms
I0324 16:16:57.751263   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 41.047661ms
I0324 16:16:57.792221   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 40.864859ms
I0324 16:16:57.832802   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 40.507304ms
I0324 16:16:57.883159   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 50.294436ms
I0324 16:16:57.923688   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 40.456835ms
I0324 16:16:57.965327   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 41.57613ms
I0324 16:16:58.005833   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 40.469677ms
I0324 16:16:58.046767   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 40.875769ms
I0324 16:16:58.088085   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 41.261827ms
I0324 16:16:58.133986   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 45.821033ms
I0324 16:16:58.174532   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 40.465473ms
I0324 16:16:58.217704   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 43.092271ms
I0324 16:16:58.258790   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 41.009155ms
I0324 16:16:58.303272   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 44.404946ms
I0324 16:16:58.343977   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 40.644881ms
I0324 16:16:58.384792   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 40.7747ms
I0324 16:16:58.425169   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 40.317656ms
I0324 16:16:58.466302   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 41.060059ms
I0324 16:16:58.507386   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 41.051664ms
I0324 16:16:58.548359   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 40.93809ms
I0324 16:16:58.588894   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 40.499528ms
I0324 16:16:58.629708   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 40.788619ms
I0324 16:16:58.674847   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 45.111125ms
I0324 16:16:58.716117   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 41.215871ms
I0324 16:16:58.757224   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 41.003277ms
I0324 16:16:58.797989   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 40.691614ms
I0324 16:16:58.961814   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 163.784435ms
I0324 16:16:59.002299   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 40.402938ms
I0324 16:16:59.042662   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 40.293538ms
I0324 16:16:59.082964   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 40.263247ms
I0324 16:16:59.123346   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 40.328564ms
I0324 16:16:59.164144   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 40.708252ms
I0324 16:16:59.204111   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 39.892719ms
I0324 16:16:59.244734   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 40.582848ms
I0324 16:16:59.294545   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 49.735649ms
I0324 16:16:59.334578   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 39.971208ms
I0324 16:16:59.378426   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 43.812878ms
I0324 16:16:59.418357   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 39.891026ms
I0324 16:16:59.462802   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 44.395118ms
I0324 16:16:59.504594   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 41.724341ms
I0324 16:16:59.544465   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 39.840592ms
I0324 16:16:59.585006   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 40.484731ms
I0324 16:16:59.625627   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 40.543641ms
I0324 16:16:59.667172   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 41.436336ms
I0324 16:16:59.707262   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 40.02222ms
I0324 16:16:59.747698   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 40.377512ms
I0324 16:16:59.788898   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 41.157729ms
I0324 16:16:59.829360   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 40.38701ms
I0324 16:16:59.870235   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 40.798787ms
I0324 16:16:59.910593   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 40.280259ms
I0324 16:16:59.950570   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 39.90202ms
I0324 16:16:59.990835   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 40.19105ms
I0324 16:17:00.031195   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 40.286255ms
I0324 16:17:00.071106   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 39.855511ms
I0324 16:17:00.110903   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 39.742696ms
I0324 16:17:00.157881   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 46.898856ms
I0324 16:17:00.197928   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 39.96792ms
I0324 16:17:00.237801   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 39.804816ms
I0324 16:17:00.277627   10550 serve_hostnames.go:247] Proxy call in namespace serve-hostnames-3700 took 39.785824ms
I0324 16:17:00.277698   10550 serve_hostnames.go:256] serve-hostname-1-0: 30  
I0324 16:17:00.277732   10550 serve_hostnames.go:256] serve-hostname-3-0: 39  
I0324 16:17:00.277757   10550 serve_hostnames.go:256] serve-hostname-0-0: 31  
W0324 16:17:00.277770   10550 serve_hostnames.go:263] No response from pod serve-hostname-2-0 on node kubernetes-minion-d0yo.c.kubernetes-satnam.internal at iteration 0
I0324 16:17:00.277790   10550 serve_hostnames.go:267] Iteration 0 took 4.270043522s for 100 queries (23.42 QPS)
I0324 16:17:00.277812   10550 serve_hostnames.go:180] Cleaning up pods
I0324 16:17:00.453097   10550 serve_hostnames.go:128] Cleaning up service serve-hostnames-3700/server-hostnames

roberthbailey
roberthbailey reviewed Mar 25, 2015
View reviewed changes
test/soak/serve_hostnames/serve_hostnames.go
},
})
})
glog.Infof("Service create %s/server-hostnames took %v", ns, time.Since(t))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be outside the loop? Otherwise it will be printed on every failed attempt as well as successful attempts.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I want to print the failed attempts. In the next iteration I plan to add a --verbose flag to make the default output quieter.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. You're trying to measure the latency of the create call here. When you add the verbose flag, you might also consider making the output reflect whether the call was successful or not.

@roberthbailey
Copy link
Contributor

Lots of nits, overall looks good.

@satnam6502
Copy link
Contributor Author

Thank you! PTAL.
In future iterations I will add a README.md explaining how to use this program with some representative output and I will add flags to make the output quieter.

@satnam6502
Copy link
Contributor Author

Oh, I also plan to add an option to use an external load balancer. That way we can compare proxy vs. ELB performance.

@roberthbailey
Copy link
Contributor

LGTM. Will merge on green.

@roberthbailey roberthbailey added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 25, 2015
rjnagal added a commit that referenced this pull request Mar 25, 2015
@rjnagal
Merge pull request #5881 from satnam6502/serve_hostnames
35b2c5c 
Change defaults, retry on errors and report stats for serve_hostnames
@rjnagal rjnagal merged commit 35b2c5c into kubernetes:master Mar 25, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@roberthbailey roberthbailey

Labels
lgtm "Looks good to me", indicates that a PR is ready to be merged.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@satnam6502 @roberthbailey @googlebot @rjnagal