Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

mutating webhook: audit log mutation existence and patch #77824

Merged
merged 4 commits into from
Aug 24, 2019
Merged

mutating webhook: audit log mutation existence and patch #77824

merged 4 commits into from
Aug 24, 2019

Conversation

roycaihw
Copy link
Member

@roycaihw roycaihw commented May 13, 2019 •
edited
Loading

Issue:

Mutating webhooks can silently mutate objects and cause API requests to fail validation (#62666). Currently when a request goes through admission chain, kube-apiserver doesn’t log / store trace of 1. which mutating webhooks mutated the request; 2. what was changed in the request.

Moreover, when a mutated request fails validation, the validation error message doesn’t necessarily reveal the actual cause (mutating webhooks) for the request to be invalid (#65569 (comment)). This makes it hard to debug a Kubernetes cluster, when (misconfigured) mutating webhooks silently invalid valid requests and put the cluster in undesired states.

What this PR does:

This PR records name of mutating webhooks in audit log to help answer the question "which mutating webhooks mutated the request".

Does this PR introduce a user-facing change?:

Audit events now log the existence and patch of mutating webhooks. 
* At Metadata audit level or higher, an annotation with key "mutation.webhook.admission.k8s.io/round_{round idx}_index_{order idx}" gets logged with JSON payload indicating a webhook gets invoked for given request and whether it mutated the object or not.
* At Request audit level or higher, an annotation with key "patch.webhook.admission.k8s.io/round_{round idx}_index_{order idx}" get logged with the JSON payload logging the patch sent by a webhook for given request.

/sig api-machinery

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-kind Indicates a PR lacks a `kind/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels May 13, 2019
@k8s-ci-robot k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels May 13, 2019
@fedebongio
Copy link
Contributor

/cc @tallclair
could you take a look to this one please?

@k8s-ci-robot k8s-ci-robot requested a review from tallclair May 13, 2019 21:18
@roycaihw
Copy link
Member Author

roycaihw commented May 13, 2019 •
edited
Loading

/cc @caesarxuchao

I'm adding an e2e test use integration test instead

@k8s-ci-robot k8s-ci-robot requested a review from caesarxuchao May 13, 2019 21:20
@roycaihw
Copy link
Member Author

/retest

@sttts
Copy link
Contributor

sttts commented May 27, 2019

Is this blocked on anything?

@k8s-ci-robot k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. sig/auth Categorizes an issue or PR as relevant to SIG Auth. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels May 28, 2019
@sttts
Copy link
Contributor

sttts commented May 29, 2019

staging/src/k8s.io/apiserver/pkg/audit/request.go:230:28: "presist" is a misspelling of "persist"

@k8s-ci-robot k8s-ci-robot removed the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label May 29, 2019
@liggitt
Copy link
Member

liggitt commented Aug 21, 2019

/hold

sorry for the hold, json construction needs a slight rework so we don't build potentially unsafe json

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 21, 2019
@k8s-ci-robot k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 23, 2019
@liggitt
Copy link
Member

liggitt commented Aug 23, 2019

lgtm, go ahead and squash fixup commits

@roycaihw
Copy link
Member Author

squashed. Thanks for the review

@roycaihw
Copy link
Member Author

/retest

@liggitt
Copy link
Member

liggitt commented Aug 23, 2019

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 23, 2019
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: liggitt, roycaihw, sttts

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@roycaihw
Copy link
Member Author

/hold cancel

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 23, 2019
@roycaihw
Copy link
Member Author

/retest

1 similar comment
@roycaihw
Copy link
Member Author

/retest

@roycaihw roycaihw mentioned this pull request Aug 23, 2019
Merged
@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Aug 24, 2019 •
edited
Loading

@roycaihw: The following test failed, say /retest to rerun them all:

Test name Commit Details Rerun command
pull-kubernetes-conformance-kind-ipv6 98ad20c link /test pull-kubernetes-conformance-kind-ipv6

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@k8s-ci-robot k8s-ci-robot merged commit e2f57be into kubernetes:master Aug 24, 2019
@tedyu
Copy link
Contributor

tedyu commented Aug 24, 2019

@roycaihw
If you think the above two comments make sense, I can send out a PR.

Thanks

@roycaihw
Copy link
Member Author

/area admission-control

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tedyu tedyu tedyu left review comments

@jpbetz jpbetz jpbetz left review comments

@sttts sttts sttts left review comments

@liggitt liggitt liggitt left review comments

@smarterclayton smarterclayton Awaiting requested review from smarterclayton

@tallclair tallclair Awaiting requested review from tallclair

@caesarxuchao caesarxuchao Awaiting requested review from caesarxuchao

Assignees

@jpbetz jpbetz

@liggitt liggitt

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/admission-control area/apiserver area/test cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API lgtm "Looks good to me", indicates that a PR is ready to be merged. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. sig/auth Categorizes an issue or PR as relevant to SIG Auth. sig/testing Categorizes an issue or PR as relevant to SIG Testing. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
Milestone
v1.16
Development

Successfully merging this pull request may close these issues.
9 participants
@roycaihw @fedebongio @sttts @fejta-bot @krmayankk @jpbetz @liggitt @k8s-ci-robot @tedyu