Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kube-proxy appears to have some bad long-tail latency issues #3734

Closed
brendandburns opened this issue Jan 22, 2015 · 1 comment
Closed

kube-proxy appears to have some bad long-tail latency issues #3734

brendandburns opened this issue Jan 22, 2015 · 1 comment
Labels
priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done. sig/network Categorizes an issue or PR as relevant to SIG Network. sig/scalability Categorizes an issue or PR as relevant to SIG Scalability.

Comments

@brendandburns
Copy link
Contributor

using redis-benchmark as a test:

with the proxy:
====== INCR ======
10000 requests completed in 0.67 seconds
50 parallel clients
3 bytes payload
keep alive: 1

1.16% <= 1 milliseconds
8.60% <= 2 milliseconds
72.41% <= 3 milliseconds
95.61% <= 4 milliseconds
98.83% <= 5 milliseconds
99.02% <= 6 milliseconds
99.25% <= 7 milliseconds
99.28% <= 8 milliseconds
99.35% <= 9 milliseconds
99.54% <= 10 milliseconds
99.58% <= 11 milliseconds
99.59% <= 13 milliseconds
99.60% <= 20 milliseconds
99.61% <= 21 milliseconds
99.62% <= 23 milliseconds
99.63% <= 25 milliseconds
99.64% <= 27 milliseconds
99.65% <= 29 milliseconds
99.66% <= 31 milliseconds
99.67% <= 32 milliseconds
99.68% <= 34 milliseconds
99.69% <= 35 milliseconds
99.70% <= 39 milliseconds
99.71% <= 40 milliseconds
99.72% <= 42 milliseconds
99.73% <= 43 milliseconds
99.74% <= 49 milliseconds
99.75% <= 52 milliseconds
99.76% <= 53 milliseconds
99.77% <= 55 milliseconds
99.78% <= 57 milliseconds
99.79% <= 60 milliseconds
99.80% <= 61 milliseconds
99.81% <= 64 milliseconds
99.82% <= 66 milliseconds
99.83% <= 68 milliseconds
99.84% <= 70 milliseconds
99.85% <= 73 milliseconds
99.86% <= 74 milliseconds
99.87% <= 77 milliseconds
99.88% <= 80 milliseconds
99.89% <= 82 milliseconds
99.90% <= 85 milliseconds
99.91% <= 89 milliseconds
99.92% <= 92 milliseconds
99.93% <= 95 milliseconds
99.94% <= 98 milliseconds
99.95% <= 100 milliseconds
99.96% <= 103 milliseconds
99.97% <= 107 milliseconds
99.98% <= 110 milliseconds
99.99% <= 113 milliseconds
100.00% <= 116 milliseconds
14880.95 requests per second

without:
====== INCR ======
10000 requests completed in 0.43 seconds
50 parallel clients
3 bytes payload
keep alive: 1

0.20% <= 1 milliseconds
67.35% <= 2 milliseconds
94.11% <= 3 milliseconds
97.21% <= 4 milliseconds
98.23% <= 5 milliseconds
99.04% <= 6 milliseconds
99.51% <= 14 milliseconds
99.82% <= 15 milliseconds
100.00% <= 15 milliseconds
23255.81 requests per second

We're basically at the same point for the 99%, but past that the proxy falls off a cliff. Would be interesting to understand why.

@roberthbailey roberthbailey added the sig/scalability Categorizes an issue or PR as relevant to SIG Scalability. label Jan 22, 2015
@bgrant0607 bgrant0607 added the sig/network Categorizes an issue or PR as relevant to SIG Network. label Jan 23, 2015
@goltermann goltermann added the priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done. label Jan 28, 2015
@thockin
Copy link
Member

thockin commented Jan 28, 2015

Dup of #3760

@thockin thockin closed this as completed Jan 28, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done. sig/network Categorizes an issue or PR as relevant to SIG Network. sig/scalability Categorizes an issue or PR as relevant to SIG Scalability.
Projects
None yet
Development

No branches or pull requests

5 participants