Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kubernetes-e2e-gci-gke-slow: broken test run #36266

Closed
k8s-github-robot opened this issue Nov 4, 2016 · 4 comments
Closed

kubernetes-e2e-gci-gke-slow: broken test run #36266

k8s-github-robot opened this issue Nov 4, 2016 · 4 comments
Assignees
@fejta @bprashanth
Labels
area/test-infra kind/flake Categorizes issue or PR as related to a flaky test. priority/backlog Higher priority than priority/awaiting-more-evidence.

Comments

@k8s-github-robot
Copy link

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-slow/1432/

Multiple broken tests:

Failed: [k8s.io] Networking [k8s.io] Granular Checks: Services [Slow] should function for pod-Service: http {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:96
Nov  4 15:25:20.483: Failed to find expected endpoints:
Tries 0
Command curl -q -s 'http://10.180.1.19:8080/dial?request=hostName&protocol=http&host=104.154.187.93&port=30865&tries=1'
retrieved map[]
expected map[netserver-0:{} netserver-1:{} netserver-2:{}]

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:206

Issues about this test specifically: #36178

Failed: [k8s.io] Networking [k8s.io] Granular Checks: Services [Slow] should update nodePort: http [Slow] {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:175
Nov  4 15:21:05.650: Failed to find expected endpoints:
Tries 0
Command timeout -t 15 curl -q -s --connect-timeout 1 --max-time 10 http://104.154.187.93:30839/hostName
retrieved map[]
expected map[netserver-1:{} netserver-2:{} netserver-0:{}]

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:257

Issues about this test specifically: #33730

Failed: [k8s.io] Networking [k8s.io] Granular Checks: Services [Slow] should function for node-Service: http {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:114
Nov  4 15:19:17.156: Failed to find expected endpoints:
Tries 0
Command timeout -t 15 curl -q -s --connect-timeout 1 --max-time 10 http://104.154.187.93:32665/hostName
retrieved map[]
expected map[netserver-0:{} netserver-1:{} netserver-2:{}]

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:257

Issues about this test specifically: #32684

Failed: [k8s.io] Networking [k8s.io] Granular Checks: Services [Slow] should function for endpoint-Service: http {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:132
Nov  4 15:19:25.297: Failed to find expected endpoints:
Tries 0
Command curl -q -s 'http://10.180.2.6:8080/dial?request=hostName&protocol=http&host=104.154.187.93&port=30950&tries=1'
retrieved map[]
expected map[netserver-0:{} netserver-1:{} netserver-2:{}]

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:206

Issues about this test specifically: #34104

Previous issues for this suite: #33597 #33989 #34665
@k8s-github-robot k8s-github-robot added kind/flake Categorizes issue or PR as related to a flaky test. priority/backlog Higher priority than priority/awaiting-more-evidence. area/test-infra labels Nov 4, 2016
@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-slow/1434/

Multiple broken tests:

Failed: [k8s.io] Networking [k8s.io] Granular Checks: Services [Slow] should function for node-Service: http {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:114
Nov  4 16:36:42.730: Failed to find expected endpoints:
Tries 0
Command timeout -t 15 curl -q -s --connect-timeout 1 --max-time 10 http://10.183.252.31:80/hostName
retrieved map[]
expected map[netserver-0:{} netserver-1:{} netserver-2:{}]

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:257

Issues about this test specifically: #32684

Failed: [k8s.io] Networking [k8s.io] Granular Checks: Services [Slow] should function for pod-Service: http {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:96
Nov  4 16:37:24.089: Failed to find expected endpoints:
Tries 0
Command curl -q -s 'http://10.180.2.13:8080/dial?request=hostName&protocol=http&host=10.183.249.13&port=80&tries=1'
retrieved map[netserver-1:{} netserver-2:{}]
expected map[netserver-0:{} netserver-1:{} netserver-2:{}]

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:206

Issues about this test specifically: #36178

Failed: [k8s.io] Networking [k8s.io] Granular Checks: Services [Slow] should update nodePort: udp [Slow] {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:187
Nov  4 16:38:12.327: Failed to find expected endpoints:
Tries 0
Command echo 'hostName' | timeout -t 2 nc -w 1 -u 104.154.144.105 30846
retrieved map[]
expected map[netserver-2:{} netserver-0:{} netserver-1:{}]

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:257

Issues about this test specifically: #33285

Failed: [k8s.io] Networking [k8s.io] Granular Checks: Services [Slow] should function for node-Service: udp {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:123
Nov  4 17:04:20.879: Failed to find expected endpoints:
Tries 0
Command echo 'hostName' | timeout -t 2 nc -w 1 -u 10.183.250.243 90
retrieved map[]
expected map[netserver-0:{} netserver-1:{} netserver-2:{}]

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:257

Issues about this test specifically: #36271

Failed: [k8s.io] Networking [k8s.io] Granular Checks: Services [Slow] should update nodePort: http [Slow] {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:175
Nov  4 16:34:50.968: Failed to find expected endpoints:
Tries 0
Command timeout -t 15 curl -q -s --connect-timeout 1 --max-time 10 http://104.154.144.105:30134/hostName
retrieved map[]
expected map[netserver-2:{} netserver-0:{} netserver-1:{}]

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:257

Issues about this test specifically: #33730

@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-slow/1435/

Multiple broken tests:

Failed: [k8s.io] Networking [k8s.io] Granular Checks: Services [Slow] should function for endpoint-Service: udp {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:141
Nov  4 17:15:08.057: Failed to find expected endpoints:
Tries 0
Command curl -q -s 'http://10.180.0.7:8080/dial?request=hostName&protocol=udp&host=104.154.144.105&port=30658&tries=1'
retrieved map[]
expected map[netserver-2:{} netserver-0:{} netserver-1:{}]

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:206

Issues about this test specifically: #34064

Failed: [k8s.io] Networking [k8s.io] Granular Checks: Services [Slow] should function for node-Service: udp {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:123
Nov  4 17:17:43.038: Failed to find expected endpoints:
Tries 0
Command echo 'hostName' | timeout -t 2 nc -w 1 -u 10.183.250.15 90
retrieved map[]
expected map[netserver-2:{} netserver-0:{} netserver-1:{}]

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:257

Issues about this test specifically: #36271

Failed: [k8s.io] Networking [k8s.io] Granular Checks: Services [Slow] should update nodePort: http [Slow] {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:175
Nov  4 17:15:57.767: Failed to find expected endpoints:
Tries 0
Command timeout -t 15 curl -q -s --connect-timeout 1 --max-time 10 http://104.154.144.105:31823/hostName
retrieved map[]
expected map[netserver-2:{} netserver-0:{} netserver-1:{}]

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:257

Issues about this test specifically: #33730

Failed: [k8s.io] Networking [k8s.io] Granular Checks: Services [Slow] should update nodePort: udp [Slow] {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:187
Nov  4 17:15:33.349: Failed to find expected endpoints:
Tries 0
Command echo 'hostName' | timeout -t 2 nc -w 1 -u 104.154.144.105 31928
retrieved map[]
expected map[netserver-0:{} netserver-1:{} netserver-2:{}]

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:257

Issues about this test specifically: #33285

Failed: [k8s.io] Networking [k8s.io] Granular Checks: Services [Slow] should function for pod-Service: http {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:96
Nov  4 17:14:43.485: Failed to find expected endpoints:
Tries 0
Command curl -q -s 'http://10.180.0.10:8080/dial?request=hostName&protocol=http&host=104.154.144.105&port=30340&tries=1'
retrieved map[]
expected map[netserver-0:{} netserver-1:{} netserver-2:{}]

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:206

Issues about this test specifically: #36178

Failed: [k8s.io] Networking [k8s.io] Granular Checks: Services [Slow] should function for node-Service: http {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:114
Nov  4 17:14:55.011: Failed to find expected endpoints:
Tries 0
Command timeout -t 15 curl -q -s --connect-timeout 1 --max-time 10 http://10.183.245.237:80/hostName
retrieved map[]
expected map[netserver-0:{} netserver-1:{} netserver-2:{}]

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:257

Issues about this test specifically: #32684

@bprashanth bprashanth self-assigned this Nov 5, 2016
@bprashanth
Copy link
Contributor

Looks like kube-proxy is reacting to endpoint changes slower than we expect. I'm uploading a temp fix till we investigate why it seems slower, but it's not slower by that much (O(5 seconds)). This might be an effet of cpu pressure on the node from another addon (heapster is the main suspect at the moment).

This was referenced Nov 5, 2016
Closed
Merged
@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-slow/1444/

Multiple broken tests:

Failed: [k8s.io] Networking [k8s.io] Granular Checks: Services [Slow] should update nodePort: udp [Slow] {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:187
Nov  4 23:36:19.405: Failed to find expected endpoints:
Tries 0
Command echo 'hostName' | timeout -t 2 nc -w 1 -u 104.198.185.171 31432
retrieved map[]
expected map[netserver-0:{} netserver-1:{} netserver-2:{}]

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:257

Issues about this test specifically: #33285

Failed: [k8s.io] Networking [k8s.io] Granular Checks: Services [Slow] should function for node-Service: http {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:114
Nov  4 23:07:30.742: Failed to find expected endpoints:
Tries 0
Command timeout -t 15 curl -q -s --connect-timeout 1 --max-time 10 http://104.198.185.171:30196/hostName
retrieved map[]
expected map[netserver-0:{} netserver-1:{} netserver-2:{}]

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:257

Issues about this test specifically: #32684 #36278

Failed: [k8s.io] Networking [k8s.io] Granular Checks: Services [Slow] should update nodePort: http [Slow] {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:175
Nov  4 23:07:18.533: Failed to find expected endpoints:
Tries 0
Command timeout -t 15 curl -q -s --connect-timeout 1 --max-time 10 http://104.198.185.171:30011/hostName
retrieved map[]
expected map[netserver-0:{} netserver-1:{} netserver-2:{}]

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:257

Issues about this test specifically: #33730

Failed: [k8s.io] Networking [k8s.io] Granular Checks: Services [Slow] should function for node-Service: udp {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:123
Nov  4 23:07:58.415: Failed to find expected endpoints:
Tries 0
Command echo 'hostName' | timeout -t 2 nc -w 1 -u 10.183.254.14 90
retrieved map[netserver-2:{} netserver-1:{}]
expected map[netserver-1:{} netserver-2:{} netserver-0:{}]

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:257

Issues about this test specifically: #36271

Failed: [k8s.io] Networking [k8s.io] Granular Checks: Services [Slow] should function for endpoint-Service: http {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:132
Nov  4 23:07:39.075: Failed to find expected endpoints:
Tries 0
Command curl -q -s 'http://10.180.0.9:8080/dial?request=hostName&protocol=http&host=104.198.185.171&port=31915&tries=1'
retrieved map[]
expected map[netserver-0:{} netserver-1:{} netserver-2:{}]

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:206

Issues about this test specifically: #34104

@k8s-github-robot k8s-github-robot mentioned this issue Nov 6, 2016
Closed
@MrHohn MrHohn mentioned this issue Nov 7, 2016
Merged
k8s-github-robot pushed a commit that referenced this issue Nov 7, 2016
Merge pull request #36332 from MrHohn/kube-proxy-ratelimit-fix
82c488b 
Automatic merge from submit-queue

Default kube-proxy to the old behavior for proxier sync.

Fix #36281.

This PR defaults `minSyncPeriod` to 0 and makes kube-proxy fall back to the old behavior to fix the immediate problem #36266.

@bprashanth
@tanjunchen tanjunchen mentioned this issue Feb 25, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@fejta fejta

@bprashanth bprashanth

Labels
area/test-infra kind/flake Categorizes issue or PR as related to a flaky test. priority/backlog Higher priority than priority/awaiting-more-evidence.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@fejta @bprashanth @k8s-github-robot