Serve OpenAPI spec with single /openapi/v2 endpoint #59293

roycaihw · 2018-02-02T23:36:23Z

What this PR does / why we need it:
We are deprecating format-separated endpoints (/swagger.json, /swagger-2.0.0.json, /swagger-2.0.0.pb-v1, /swagger-2.0.0.pb-v1.gz) for OpenAPI spec, and switching to a single /openapi/v2 endpoint in Kubernetes 1.10. The design doc and deprecation process are tracked at: https://docs.google.com/document/d/19lEqE9lc4yHJ3WJAJxS_G7TcORIJXGHyq3wpwcH28nU

Requested format is specified by setting HTTP headers

header	possible values
Accept	`application/json`, `application/com.github.proto-openapi.spec.v2@v1.0+protobuf`
Accept-Encoding	`gzip`

This PR changes dynamic_client (and kubectl as a result) to use the new endpoint. The old endpoints will remain in 1.10, 1.11, 1.12 and 1.13, and get removed in 1.14.

Examples:

previous	now
GET /swagger.json	GET /openapi/v2 Accept: application/json
GET /swagger-2.0.0.pb-v1	GET /openapi/v2 Accept: application/com.github.proto-openapi.spec.v2@v1.0+protobuf
GET /swagger-2.0.0.pb-v1.gz	GET /openapi/v2 Accept: application/com.github.proto-openapi.spec.v2@v1.0+protobuf Accept-Encoding: gzip

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Special notes for your reviewer:

Release note:

action required: Deprecate format-separated endpoints for OpenAPI spec. Please use single `/openapi/v2` endpoint instead.

/sig api-machinery

lavalamp · 2018-02-05T21:06:10Z

/assign @yliaog
/assign @mbohlool

roycaihw · 2018-02-05T23:49:46Z

plugin/pkg/auth/authorizer/rbac/bootstrappolicy/policy.go

@@ -175,6 +175,7 @@ func ClusterRoles() []rbac.ClusterRole {
 					// do not expand this pattern for openapi discovery docs
 					// move to a single openapi endpoint that takes accept/accept-encoding headers
 					"/swagger.json", "/swagger-2.0.0.pb-v1",
+					"/openapi", "/openapi/*",


Saving the place for /openapi/v3 in future. Also, we will add /openapi (which returns list of supported versions) as a followup.

liggitt · 2018-02-06T04:23:37Z

vendor/k8s.io/kube-openapi/pkg/handler/handler.go

@@ -77,6 +78,10 @@ func computeETag(data []byte) string {
 	return fmt.Sprintf("\"%X\"", sha512.Sum512(data))
 }

+// NOTE: [DEPRECATION] We will announce deprecation for format-separated endpoints for OpenAPI spec,


Switch is in 1.10

Sent out PR to correct that in kube-openapi

liggitt · 2018-02-06T04:26:02Z

vendor/k8s.io/kube-openapi/pkg/handler/handler.go

+				decipherableFormat = "application/json"
+			}
+			w.Header().Add("Vary", "Accept")
+


This needs to work with actual Accept headers, which are allowed to specify a prioritized list of accepted content types. For example, a request could indicate it accepts some unknown protobuf format, the known protobuf format, yaml, and json, and the server would be expected to return the known protobuf format

For example, see parsing/prioritization/matching in

kubernetes/staging/src/k8s.io/apiserver/pkg/endpoints/handlers/negotiation/negotiate.go

Lines 284 to 312 in 228b7d5

func NegotiateMediaTypeOptions(header string, accepted []AcceptedMediaType, endpoint EndpointRestrictions) (MediaTypeOptions, bool) {

if len(header) == 0 && len(accepted) > 0 {

return MediaTypeOptions{

Accepted: &accepted[0],

}, true

}

var candidates candidateMediaTypeSlice

clauses := goautoneg.ParseAccept(header)

for _, clause := range clauses {

for i := range accepted {

accepts := &accepted[i]

switch {

case clause.Type == accepts.Type && clause.SubType == accepts.SubType,

clause.Type == accepts.Type && clause.SubType == "*",

clause.Type == "*" && clause.SubType == "*":

candidates = append(candidates, candidateMediaType{accepted: accepts, clauses: clause})

}

}

}

for _, v := range candidates {

if retVal, ret := acceptMediaTypeOptions(v.clauses.Params, v.accepted, endpoint); ret {

return retVal, true

}

}

return MediaTypeOptions{}, false

}

Thanks! PR sent in kube-openapi repo to use that kubernetes/kube-openapi#36

liggitt · 2018-02-06T04:30:53Z

staging/src/k8s.io/client-go/discovery/discovery_client.go

@@ -329,7 +333,7 @@ func (d *DiscoveryClient) ServerVersion() (*version.Info, error) {

 // OpenAPISchema fetches the open api schema using a rest client and parses the proto.
 func (d *DiscoveryClient) OpenAPISchema() (*openapi_v2.Document, error) {
-	data, err := d.restClient.Get().AbsPath("/swagger-2.0.0.pb-v1").Do().Raw()
+	data, err := d.restClient.Get().AbsPath("/openapi/v2").SetHeader("Accept", mimePb).Do().Raw()


Cannot change this until 1.11 to maintain one version skew compatibility in kubectl

we can do that or we can try new endpoint first and then fallback on the old one. Pro: we have new endpoints in one version and if there is any problem, we would figure it out before completely remove the old endpoint. Con: new clients will have a little more latency to talk to old clusters.

I was working on the fallback logic after I sent out the PR. Thanks for pointing that out :) Updated.

liggitt · 2018-02-06T04:35:18Z

vendor/k8s.io/kube-openapi/pkg/handler/handler.go

+		"application/com.github.proto-openapi.spec.v2@v1.0+protobuf": o.getSwaggerPbBytes,
+	}
+
+	handler.Handle(servePath, gziphandler.GzipHandler(http.HandlerFunc(


Does this handle varying on Accept-Encoding as well?

Yes, according to

kubernetes/vendor/github.com/NYTimes/gziphandler/gzip.go

Line 245 in f821a54

if acceptsGzip(r) {

mbohlool · 2018-02-06T11:25:53Z

staging/src/k8s.io/kube-aggregator/pkg/controllers/openapi/aggregator.go

+	if err != nil {
+		return err
+	}
+	return s.openAPIVersionedService.UpdateSpec(specToServe)


do we know that versionedService's updateSpec will also update the spec fo the old endpoint? We should verify that and add a comment here to document we confirmed that.

Comment added

mbohlool · 2018-02-06T11:27:08Z

staging/src/k8s.io/kube-aggregator/pkg/controllers/openapi/aggregator.go

 		return nil
 	}
 	specToServe, err := s.buildOpenAPISpec()
 	if err != nil {
 		return err
 	}
-	return s.openAPIService.UpdateSpec(specToServe)
+	err = s.openAPIService.UpdateSpec(specToServe)


I am pretty sure somewhere in aggregation, we access /swagger.json file. It should be somewhere in kube-openapi repo. Can you please verify that we updated that code to try version'ed endpoint first (with fallback to /swagger.json endpoint).

I didn't find /swagger.json being accessed in kube-openapi repo. Do you mean staging/src/k8s.io/kube-aggregator/pkg/controllers/openapi/downloader.go? Change added

mbohlool · 2018-02-06T11:31:04Z

staging/src/k8s.io/client-go/discovery/discovery_client.go

@@ -329,7 +333,7 @@ func (d *DiscoveryClient) ServerVersion() (*version.Info, error) {

 // OpenAPISchema fetches the open api schema using a rest client and parses the proto.
 func (d *DiscoveryClient) OpenAPISchema() (*openapi_v2.Document, error) {
-	data, err := d.restClient.Get().AbsPath("/swagger-2.0.0.pb-v1").Do().Raw()
+	data, err := d.restClient.Get().AbsPath("/openapi/v2").SetHeader("Accept", mimePb).Do().Raw()


we can do that or we can try new endpoint first and then fallback on the old one. Pro: we have new endpoints in one version and if there is any problem, we would figure it out before completely remove the old endpoint. Con: new clients will have a little more latency to talk to old clusters.

fejta-bot · 2018-02-24T23:55:49Z

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel comment for consistent failures.

fejta-bot · 2018-02-25T02:43:49Z

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel comment for consistent failures.

fejta-bot · 2018-02-25T05:31:49Z

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel comment for consistent failures.

fejta-bot · 2018-02-25T08:19:48Z

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel comment for consistent failures.

fejta-bot · 2018-02-25T11:07:49Z

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel comment for consistent failures.

fejta-bot · 2018-02-25T13:55:48Z

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel comment for consistent failures.

fejta-bot · 2018-02-25T16:43:48Z

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel comment for consistent failures.

fejta-bot · 2018-02-25T19:31:47Z

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel comment for consistent failures.

k8s-github-robot · 2018-02-25T20:01:35Z

/test all [submit-queue is verifying that this PR is safe to merge]

fejta-bot · 2018-02-25T22:40:47Z

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel comment for consistent failures.

k8s-github-robot · 2018-02-25T23:09:29Z

/test all [submit-queue is verifying that this PR is safe to merge]

fejta-bot · 2018-02-26T01:49:48Z

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel comment for consistent failures.

fejta-bot · 2018-02-26T04:37:49Z

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel comment for consistent failures.

fejta-bot · 2018-02-26T07:25:49Z

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel comment for consistent failures.

fejta-bot · 2018-02-26T10:13:48Z

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel comment for consistent failures.

fejta-bot · 2018-02-26T13:01:48Z

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel comment for consistent failures.

fejta-bot · 2018-02-26T15:49:48Z

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel comment for consistent failures.

fejta-bot · 2018-02-26T18:37:49Z

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel comment for consistent failures.

roycaihw · 2018-02-26T19:28:43Z

TestCRDDeletionCascading flake #59426

roycaihw · 2018-02-26T20:54:38Z

/retest

fejta-bot · 2018-02-26T23:10:49Z

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel comment for consistent failures.

fejta-bot · 2018-02-27T01:58:50Z

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel comment for consistent failures.

k8s-github-robot · 2018-02-27T02:34:25Z

/test all

Tests are more than 96 hours old. Re-running tests.

k8s-github-robot · 2018-02-27T07:47:51Z

Automatic merge from submit-queue (batch tested with PRs 60011, 59256, 59293, 60328, 60367). If you want to cherry-pick this change to another branch, please follow the instructions here.

k8s-ci-robot requested review from gmarek, jbeda, liggitt and sttts February 2, 2018 23:37

roycaihw force-pushed the openapi_endpoint branch 2 times, most recently from 63ee6d3 to bf5dc3a Compare February 5, 2018 18:48

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Feb 5, 2018

k8s-ci-robot assigned mbohlool and yliaog Feb 5, 2018

roycaihw commented Feb 5, 2018

View reviewed changes

roycaihw force-pushed the openapi_endpoint branch from fb23bf2 to 173b0ec Compare February 6, 2018 00:23

liggitt reviewed Feb 6, 2018

View reviewed changes

mbohlool reviewed Feb 6, 2018

View reviewed changes

roycaihw force-pushed the openapi_endpoint branch 2 times, most recently from 4d3ae79 to b07f6d4 Compare February 7, 2018 18:37

roycaihw force-pushed the openapi_endpoint branch from b07f6d4 to 119da39 Compare February 20, 2018 17:23

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 20, 2018

roycaihw added 2 commits February 20, 2018 09:27

Bump kube-openapi to add new openapi endpoint

8b38e08

Add new openapi endpoint in aggregator server

2eb3d04

roycaihw force-pushed the openapi_endpoint branch from 119da39 to f422dec Compare February 20, 2018 17:27

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 20, 2018

k8s-github-robot merged commit d615319 into kubernetes:master Feb 27, 2018

roycaihw deleted the openapi_endpoint branch February 27, 2018 17:53

CaoShuFeng mentioned this pull request Mar 28, 2018

Cannot update cronjob from within pod (NotAcceptable 406 error) #61805

Closed

roycaihw mentioned this pull request Apr 11, 2018

Document new OpenAPI endpoint kubernetes/website#8050

Merged

roycaihw mentioned this pull request May 22, 2018

Redirect deprecated OpenAPI endpoint to new one #64129

Closed

ChiuWang mentioned this pull request Jun 7, 2018

try not fetch old endpoint in 1.11 #64868

Closed

	func NegotiateMediaTypeOptions(header string, accepted []AcceptedMediaType, endpoint EndpointRestrictions) (MediaTypeOptions, bool) {
	if len(header) == 0 && len(accepted) > 0 {
	return MediaTypeOptions{
	Accepted: &accepted[0],
	}, true
	}

	var candidates candidateMediaTypeSlice
	clauses := goautoneg.ParseAccept(header)
	for _, clause := range clauses {
	for i := range accepted {
	accepts := &accepted[i]
	switch {
	case clause.Type == accepts.Type && clause.SubType == accepts.SubType,
	clause.Type == accepts.Type && clause.SubType == "*",
	clause.Type == "" && clause.SubType == "":
	candidates = append(candidates, candidateMediaType{accepted: accepts, clauses: clause})
	}
	}
	}

	for _, v := range candidates {
	if retVal, ret := acceptMediaTypeOptions(v.clauses.Params, v.accepted, endpoint); ret {
	return retVal, true
	}
	}

	return MediaTypeOptions{}, false
	}

Serve OpenAPI spec with single /openapi/v2 endpoint #59293

Serve OpenAPI spec with single /openapi/v2 endpoint #59293

Conversation

roycaihw commented Feb 2, 2018 • edited Loading

lavalamp commented Feb 5, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liggitt Feb 6, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fejta-bot commented Feb 24, 2018

fejta-bot commented Feb 25, 2018

fejta-bot commented Feb 25, 2018

fejta-bot commented Feb 25, 2018

fejta-bot commented Feb 25, 2018

fejta-bot commented Feb 25, 2018

fejta-bot commented Feb 25, 2018

fejta-bot commented Feb 25, 2018

k8s-github-robot commented Feb 25, 2018

fejta-bot commented Feb 25, 2018

k8s-github-robot commented Feb 25, 2018

fejta-bot commented Feb 26, 2018

fejta-bot commented Feb 26, 2018

fejta-bot commented Feb 26, 2018

fejta-bot commented Feb 26, 2018

fejta-bot commented Feb 26, 2018

fejta-bot commented Feb 26, 2018

fejta-bot commented Feb 26, 2018

roycaihw commented Feb 26, 2018

roycaihw commented Feb 26, 2018

fejta-bot commented Feb 26, 2018

fejta-bot commented Feb 27, 2018

k8s-github-robot commented Feb 27, 2018

k8s-github-robot commented Feb 27, 2018

roycaihw commented Feb 2, 2018 •

edited

Loading

liggitt Feb 6, 2018 •

edited

Loading