Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Wait for healthy extension server before registering APIService, handle ServiceUnavailable errors #58697

Merged
merged 3 commits into from
Jan 23, 2018
Merged

Wait for healthy extension server before registering APIService, handle ServiceUnavailable errors #58697

merged 3 commits into from
Jan 23, 2018

Conversation

liggitt
Copy link
Member

@liggitt liggitt commented Jan 23, 2018 •
edited
Loading

fixes #58642
followup to #58070

  • Because a registered APIService appears in discovery immediately, we should wait until the backing deployment is healthy before exposing it
  • In e2e hasRemainingContent(), add ServiceUnavailable to the types of errors we tolerate when looking for remaining content.
  • In proxy handler, return a ServiceUnavailable error if the referenced service cannot be resolved
NONE

@k8s-ci-robot k8s-ci-robot added do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Jan 23, 2018
@k8s-ci-robot k8s-ci-robot requested review from bowei and marun January 23, 2018 15:02
sttts
sttts reviewed Jan 23, 2018
View reviewed changes
test/e2e/apimachinery/aggregator.go
// kubectl get deployments -n <aggregated-api-namespace> && status == Running
// NOTE: aggregated apis should generally be set up in there own namespace (<aggregated-api-namespace>). As the test framework
// is setting up a new namespace, we are just using that.
err = framework.WaitForDeploymentComplete(client, deployment)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm. Not sure it's the source for the flakes, but it could be.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need add a framework.ExpectNoError call to ensure err is nil?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, added

@liggitt liggitt force-pushed the aggregator-e2e-fix branch from a968aab to b967d1b Compare January 23, 2018 15:20
@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 23, 2018
@k8s-ci-robot k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Jan 23, 2018
sttts
sttts reviewed Jan 23, 2018
View reviewed changes
test/e2e/framework/util.go
@@ -1200,6 +1200,10 @@ func hasRemainingContent(c clientset.Interface, clientPool dynamic.ClientPool, n
if apierrs.IsMethodNotSupported(err) || apierrs.IsNotFound(err) || apierrs.IsForbidden(err) {
continue
}
// skip unavailable servers
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe add a comment why we add this

@sttts
Copy link
Contributor

sttts commented Jan 23, 2018

Only a nit. Otherwise lgtm.

@sttts
Copy link
Contributor

sttts commented Jan 23, 2018

/cc @hzxuzhonghu

@liggitt liggitt force-pushed the aggregator-e2e-fix branch from 84e4e81 to 193aa57 Compare January 23, 2018 15:46
@k8s-ci-robot k8s-ci-robot removed the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 23, 2018
sttts
sttts reviewed Jan 23, 2018
View reviewed changes
staging/src/k8s.io/kube-aggregator/pkg/apiserver/handler_proxy.go Outdated
@@ -120,7 +120,8 @@ func (r *proxyHandler) ServeHTTP(w http.ResponseWriter, req *http.Request) {
location.Scheme = "https"
rloc, err := r.serviceResolver.ResolveEndpoint(handlingInfo.serviceNamespace, handlingInfo.serviceName)
if err != nil {
http.Error(w, fmt.Sprintf("missing route (%s)", err.Error()), http.StatusInternalServerError)
glog.Errorf("error resolving %s/%s: %v", handlingInfo.serviceName, handlingInfo.serviceNamespace, err)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ns/name

@liggitt liggitt force-pushed the aggregator-e2e-fix branch from 193aa57 to f045b00 Compare January 23, 2018 15:48
@liggitt liggitt force-pushed the aggregator-e2e-fix branch from f045b00 to 91ba8c3 Compare January 23, 2018 16:01
@k8s-ci-robot k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Jan 23, 2018
@liggitt liggitt changed the title Wait for healthy extension server before registering APIService Wait for healthy extension server before registering APIService, handle ServiceUnavailable errors Jan 23, 2018
@sttts
Copy link
Contributor

sttts commented Jan 23, 2018

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 23, 2018
@sttts
Copy link
Contributor

sttts commented Jan 23, 2018

/assign deads2k

@liggitt liggitt added queue/fix kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. labels Jan 23, 2018
@deads2k
Copy link
Contributor

deads2k commented Jan 23, 2018

/approve

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: deads2k, liggitt, sttts

Associated issue: #58642

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 23, 2018
@k8s-github-robot
Copy link

/test all [submit-queue is verifying that this PR is safe to merge]

@Random-Liu
Copy link
Member

@liggitt Thanks for the fix! :)

@liggitt
Copy link
Member Author

liggitt commented Jan 23, 2018

/retest

@k8s-github-robot
Copy link

Automatic merge from submit-queue (batch tested with PRs 58697, 58658, 58676, 58674). If you want to cherry-pick this change to another branch, please follow the instructions here.

@k8s-github-robot k8s-github-robot merged commit 8d62044 into kubernetes:master Jan 23, 2018
@k8s-ci-robot
Copy link
Contributor

@liggitt: The following test failed, say /retest to rerun them all:

Test name Commit Details Rerun command
pull-kubernetes-e2e-kops-aws 91ba8c3 link /test pull-kubernetes-e2e-kops-aws

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@Random-Liu Random-Liu mentioned this pull request Jan 23, 2018
Closed
@liggitt liggitt deleted the aggregator-e2e-fix branch January 26, 2018 04:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sttts sttts sttts left review comments

@weekface weekface weekface left review comments

@bowei bowei Awaiting requested review from bowei

@marun marun Awaiting requested review from marun

@hzxuzhonghu hzxuzhonghu Awaiting requested review from hzxuzhonghu

Assignees

@sttts sttts

@deads2k deads2k

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. lgtm "Looks good to me", indicates that a PR is ready to be merged. release-note-none Denotes a PR that doesn't merit a release note. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
7 participants
@liggitt @sttts @deads2k @k8s-ci-robot @k8s-github-robot @Random-Liu @weekface