Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

release-1.8 GKE test failure "Services should work after restarting apiserver" #55244

Closed
jpbetz opened this issue Nov 7, 2017 · 5 comments
Closed

release-1.8 GKE test failure "Services should work after restarting apiserver" #55244

jpbetz opened this issue Nov 7, 2017 · 5 comments
Labels
milestone/incomplete-labels priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. release-blocker
Milestone
v1.8

Comments

@jpbetz
Copy link
Contributor

jpbetz commented Nov 7, 2017

In the release1.8 blocking test suite. "[sig-network] Services should work after restarting apiserver [Disruptive]" has started failing regularly with the error:

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/service.go:434
Nov  4 02:53:35.323: error restarting apiserver: error running gcloud [container clusters --project=k8s-gke-upg-c1-4-g1-6-up-clu --zone=us-central1-b upgrade e2e-856 --master --cluster-version=1.8.3-beta.0.88+dc6c4c5dcebb7e --quiet]; got error exit status 1, stdout "", stderr "Upgrading e2e-856...\n.....done.\nERROR: (gcloud.container.clusters.upgrade) Operation [<Operation\n endTime: u'2017-11-04T02:53:29.555870594Z'\n name: u'operation-1509764009232-95e55dde'\n operationType: OperationTypeValueValuesEnum(UPGRADE_MASTER, 3)\n selfLink: u'https://test-container.sandbox.googleapis.com/v1/projects/433788295746/zones/us-central1-b/operations/operation-1509764009232-95e55dde'\n startTime: u'2017-11-04T02:53:29.232057635Z'\n status: StatusValueValuesEnum(DONE, 3)\n statusMessage: u'Patch failed'\n targetLink: u'https://test-container.sandbox.googleapis.com/v1/projects/433788295746/zones/us-central1-b/clusters/e2e-856'\n zone: u'us-central1-b'>] finished with error: Patch failed\n"
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/service.go:414

The first occurance I can find is in build 841. But it has been failing more consistently since around build 856.
@jpbetz jpbetz added release-blocker sig/network Categorizes an issue or PR as relevant to SIG Network. labels Nov 7, 2017
@jpbetz jpbetz added this to the v1.8 milestone Nov 7, 2017
@jpbetz
Copy link
Contributor Author

jpbetz commented Nov 7, 2017

cc @tallclair

@jpbetz jpbetz mentioned this issue Nov 7, 2017
Merged
@jpbetz jpbetz added the priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. label Nov 7, 2017
@k8s-github-robot
Copy link

[MILESTONENOTIFIER] Milestone Labels Incomplete

@jpbetz

Action required: This issue requires label changes.

kind: Must specify exactly one of kind/bug, kind/cleanup or kind/feature.

Help

@jpbetz jpbetz mentioned this issue Nov 7, 2017
Merged
@bboreham
Copy link
Contributor

bboreham commented Nov 7, 2017

That's a gcloud error; the code only wants to restart the api-server but from the comments there is no way to do that on GKE so it does an "upgrade" to the same version.

The error is nothing to do with networking.

@enisoc
Copy link
Member

enisoc commented Nov 7, 2017

This may have the same root cause as #55189. We're having problems with almost all GKE e2e test due to clusters getting stuck during startup.

@enisoc enisoc added sig/gcp and removed sig/network Categorizes an issue or PR as relevant to SIG Network. labels Nov 7, 2017
@jpbetz jpbetz mentioned this issue Nov 7, 2017
Closed
@jpbetz
Copy link
Contributor Author

jpbetz commented Nov 8, 2017 •
edited
Loading

GKE is no longer returning 'patch failed' and tests are passing agian.

@jpbetz jpbetz closed this as completed Nov 8, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
milestone/incomplete-labels priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. release-blocker
Projects
None yet
Milestone
v1.8
Development

No branches or pull requests
4 participants
@jpbetz @enisoc @bboreham @k8s-github-robot