Add e2e test for API coverage #57171

roycaihw · 2017-12-14T02:15:38Z

What this PR does / why we need it:
Use dynamic client to simply test CRUD operations on all expected resources. Most of the YAML files are collected from kubernetes code base and kubernetes.io. Currently the test covers 406 operations (test/e2e/testing-manifests/apiresource/resources.csv) out of 702 operations (test/e2e/testing-manifests/apiresource/resources_all.csv).

The csv files are in the format of group, version, resource, namespaced, verb. The remaining resources are not tested yet because either the yaml file is missing, or the resource is non-namespaced and may break e2e test. Ref test/e2e/testing-manifests/apiresource/README.md for more information.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Ref kubernetes/test-infra#5446

Special notes for your reviewer:

Release note:

NONE

/sig api-machinery
cc @erictune

roycaihw · 2017-12-15T01:33:03Z

/retest

roycaihw · 2018-02-16T00:18:13Z

/cc @erictune @fejta @mithrav

k8s-ci-robot · 2018-02-16T00:18:16Z

@roycaihw: GitHub didn't allow me to request PR reviews from the following users: mithrav.

Note that only kubernetes members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/cc @erictune @fejta @mithrav

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

fejta · 2018-02-23T23:16:58Z

/uncc
/assign @lavalamp @deads2k

Looks like apimachinery stuff?

fejta · 2018-02-23T23:17:14Z

@k8s-mirror-api-machinery-pr-reviews

fejta · 2018-02-23T23:17:33Z

@kubernetes/sig-api-machinery-pr-reviews

deads2k · 2018-03-05T14:51:13Z

Wow, this looks like a maintenance nightmare. What is is goal? Is any/most of this generated?

@liggitt @derekwaynecarr called it?

thockin · 2018-03-05T14:51:22Z

This change is

bgrant0607 · 2018-03-09T15:43:19Z

If we want to include this in conformance, one thing we'll need to figure out is how to exclude APIs that should not be covered. The existing coverage data already shows that some machinery is probing paths that should be excluded.

https://gist.github.com/oomichi/bc0044624443b4b596bdb4c83dd4ad4b

cc @kubernetes/sig-architecture-pr-reviews

roycaihw · 2018-03-30T18:43:20Z

@deads2k The goal is to use a table driven test to verify if a Kubernetes cluster exposes endpoints an end user would reasonably expect. This test doesn't verify the correct behavior of those APIs, but just checks if the endpoints exist (as a necessary but not sufficient step).

For the maintenance, may I ask if you are talking about the list of APIs or the yaml test data? The API list is generated by bringing up a cluster and using discovery API. The yaml test data is collected from the code base and kubernetes.io. It would be better if we could generate those yaml files from the schema/Go types. But if we manually maintain them, are we expecting the yaml file will be invalid (broken) for a beta/GA API frequently?

sttts · 2018-04-03T12:12:46Z

test/e2e/apimachinery/coverage.go

+	subresources resourceMap
+}
+
+var _ = SIGDescribe("Coverage", func() {


This belong into some test/e2e/coverage folder. It tests coverage of Kubernetes APIs, it's not about generic API features.

This comment still applies.

mml · 2018-05-19T00:00:21Z

Please squash. You should have time since the tests are failing.

roycaihw · 2018-05-22T18:21:31Z

@mml Squashed. PTAL

/test pull-kubernetes-e2e-kops-aws

mml · 2018-05-23T22:41:31Z

/lgtm

k8s-ci-robot · 2018-05-23T22:41:37Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: mml, roycaihw
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: cblecker

Assign the PR to them by writing /assign @cblecker in a comment when ready.

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

hack/OWNERS
test/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

roycaihw · 2018-05-23T22:53:53Z

/assign @sttts @cblecker
for approval

`hack/update-api-resource-list.sh` and `hack/verify-api-resource-list.sh` generate and verify list of api resources served by kube apiserver. `test/e2e/testing-manifests/apiresource/resources_all.csv` is the generated list of all api resources served by kube apiserver. `test/e2e/testing-manifests/apiresource/resources_whitelist.csv` is the whitelist of api verbs that we are not covering in api coverage test `test/e2e/testing-manifests/apiresource/resources.csv` is the actual list of api verbs that we test in api coverage test, which is generated based on `resources_all.csv` and `resources_whitelist.csv`

k8s-ci-robot · 2018-05-29T23:11:51Z

New changes are detected. LGTM label has been removed.

roycaihw · 2018-05-30T02:20:57Z

/test pull-kubernetes-verify

roycaihw · 2018-06-01T21:34:58Z

From a test infra perspective, the value from verifying existence of API CRUD operations doesn't match the effort of hand-crafting object yaml files that exercise API validation. Nevertheless, we should think about having better documentation & examples in our OpenAPI models and verbs. Ref #34743, https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.10/

From conformance perspective, we should focus on testing the correct API behavior on pluggable parts of the system, and increasing coverage with those tests. Ref kubernetes/test-infra#5446 (comment)

Closing

k8s-github-robot assigned bowei and shyamjvs Dec 14, 2017

roycaihw force-pushed the api_coverage_release branch from c246b6c to 9f9ec17 Compare December 14, 2017 22:46

k8s-github-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 15, 2018

roycaihw force-pushed the api_coverage_release branch from 498ed9d to d21e49e Compare February 15, 2018 22:05

roycaihw changed the title ~~[WIP] Add e2e test for API coverage~~ Add e2e test for API coverage Feb 15, 2018

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 15, 2018

k8s-github-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 15, 2018

k8s-ci-robot requested review from erictune and fejta February 16, 2018 00:18

k8s-ci-robot assigned deads2k and lavalamp Feb 23, 2018

k8s-ci-robot removed the request for review from fejta February 23, 2018 23:16

k8s-ci-robot added the sig/architecture Categorizes an issue or PR as relevant to SIG Architecture. label Mar 9, 2018

sttts reviewed Apr 3, 2018

View reviewed changes

roycaihw force-pushed the api_coverage_release branch from c7e8750 to 44426d2 Compare May 21, 2018 17:31

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 21, 2018

roycaihw force-pushed the api_coverage_release branch 2 times, most recently from ace194a to dd3d069 Compare May 22, 2018 03:51

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 23, 2018

k8s-ci-robot assigned sttts May 23, 2018

roycaihw added 3 commits May 29, 2018 13:54

Prepare API resources yaml files using bindata

3774672

Add API coverage test

bd85649

roycaihw force-pushed the api_coverage_release branch from dd3d069 to e59e43b Compare May 29, 2018 23:11

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 29, 2018

roycaihw force-pushed the api_coverage_release branch 7 times, most recently from 026f658 to 934cb31 Compare May 29, 2018 23:41

roycaihw added 2 commits May 29, 2018 17:26

Rebase and update api resource list

b05d006

Documentation

cb43d50

roycaihw force-pushed the api_coverage_release branch from 934cb31 to cb43d50 Compare May 30, 2018 00:27

roycaihw closed this Jun 1, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add e2e test for API coverage #57171

Add e2e test for API coverage #57171

roycaihw commented Dec 14, 2017 •

edited

Loading

roycaihw commented Dec 15, 2017

roycaihw commented Feb 16, 2018

k8s-ci-robot commented Feb 16, 2018

fejta commented Feb 23, 2018

fejta commented Feb 23, 2018

fejta commented Feb 23, 2018

deads2k commented Mar 5, 2018

thockin commented Mar 5, 2018

bgrant0607 commented Mar 9, 2018

roycaihw commented Mar 30, 2018

sttts Apr 3, 2018

sttts May 7, 2018

mml commented May 19, 2018

roycaihw commented May 22, 2018

mml commented May 23, 2018

k8s-ci-robot commented May 23, 2018

roycaihw commented May 23, 2018

k8s-ci-robot commented May 29, 2018

roycaihw commented May 30, 2018

roycaihw commented Jun 1, 2018

Add e2e test for API coverage #57171

Add e2e test for API coverage #57171

Conversation

roycaihw commented Dec 14, 2017 • edited Loading

roycaihw commented Dec 15, 2017

roycaihw commented Feb 16, 2018

k8s-ci-robot commented Feb 16, 2018

fejta commented Feb 23, 2018

fejta commented Feb 23, 2018

fejta commented Feb 23, 2018

deads2k commented Mar 5, 2018

thockin commented Mar 5, 2018

bgrant0607 commented Mar 9, 2018

roycaihw commented Mar 30, 2018

sttts Apr 3, 2018

Choose a reason for hiding this comment

sttts May 7, 2018

Choose a reason for hiding this comment

mml commented May 19, 2018

roycaihw commented May 22, 2018

mml commented May 23, 2018

k8s-ci-robot commented May 23, 2018

roycaihw commented May 23, 2018

k8s-ci-robot commented May 29, 2018

roycaihw commented May 30, 2018

roycaihw commented Jun 1, 2018

roycaihw commented Dec 14, 2017 •

edited

Loading