Skip to content

Commit

Permalink
feat: use hooks as a validating webhook handlers (#223)
Browse files Browse the repository at this point in the history 
feat: use hooks as a validating webhook handlers

- new binding type `kubernetesValidating`
- add documentation and example
- some ci enhancements: build dev image for amd64 only, remove -v from tests

Co-authored-by: Andrey Klimentyev <andrey.klimentyev@flant.com>
  • Loading branch information
@diafour @zuzzas
diafour and zuzzas authored Jan 12, 2021
1 parent 0db09ac commit 51b75bf
Show file tree
Hide file tree
Showing 70 changed files with 3,289 additions and 269 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/code-checks.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -39,4 +39,4 @@ jobs:
- name: codespell
run: |
pip install codespell==v1.17.1
codespell --skip=".git,go.mod,go.sum,*.log,*.gif,*.png" -L witht,eventtypes,uint,uptodate
codespell --skip=".git,go.mod,go.sum,*.log,*.gif,*.png" -L witht,eventtypes,uint,uptodate,keypair
98 changes: 98 additions & 0 deletions .github/workflows/publish-dev-amd64.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
name: Publish dev image amd64
on:
pull_request:
types: [labeled]
env:
# ':robot: build dev image: amd64' label
# not working on job level :-(
# https://github.community/t/how-to-set-and-access-a-workflow-variable/17335/3
LABEL_ID: 2648778919
# build only amd64 to speed up dev image build
BUILDX_PLATFORMS: "linux/amd64"
IMAGE_REPO: flant/shell-operator-dev

jobs:
# Empty job if PR labeled with another label.
stub:
name: Empty job to prevent workflow fail
runs-on: ubuntu-latest
if: github.event_name == 'pull_request' && github.event.label.id != 2648778919
steps:
- name: stub action
run: ": This job is used to prevent the workflow to fail when all other jobs are skipped."

# Remove label from PR.
unlabel:
name: Unlabel
runs-on: ubuntu-latest
if: github.event_name == 'pull_request' && github.event.label.id == 2648778919
steps:
- uses: actions/github-script@v3
with:
github-token: ${{secrets.GITHUB_TOKEN}}
script: |
const eventLabelName = '${{github.event.label.name}}'
const response = await github.issues.listLabelsOnIssue({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.issue.number
})
for (const label of response.data) {
if (label.name === eventLabelName) {
github.issues.removeLabel({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.issue.number,
name: eventLabelName
})
break
}
}
build_dev_image:
name: Dev image
runs-on: ubuntu-latest
if: github.event_name == 'pull_request' && github.event.label.id == 2648778919
steps:
- uses: actions/checkout@v2

- name: Prepare environment
run: |
: Setup imageTag, appVersion and dockerFile envs and build image
imageTag=${GITHUB_REF#refs/tags/}
APP_VERSION=${imageTag}
FINAL_IMAGE_NAME="${IMAGE_REPO}:${imageTag}"
: Override image name and version for dev image
# dev-feat_branch-371e2d3b-2020.02.06_18:37:42
APP_VERSION=${GITHUB_REF#refs/heads/}-${GITHUB_SHA::8}-$(date +'%Y.%m.%d_%H:%M:%S')
FINAL_IMAGE_NAME="${IMAGE_REPO}:pr${{ github.event.pull_request.number }}"
: end override
echo "FINAL_IMAGE_NAME=${FINAL_IMAGE_NAME}" >> ${GITHUB_ENV}
echo "APP_VERSION=${APP_VERSION}" >> ${GITHUB_ENV}
echo "========================================="
echo "APP_VERSION = $APP_VERSION"
echo "FINAL_IMAGE_NAME = $FINAL_IMAGE_NAME"
echo "========================================="
- name: Set up Docker Buildx
id: buildx
uses: docker/setup-buildx-action@v1
with:
version: latest

- name: Login to DockerHub
uses: docker/login-action@v1
with:
username: ${{ secrets.DOCKER_USER }}
password: ${{ secrets.DOCKER_PASS }}

- name: Build and push multi-arch image
run: |
echo "Build $FINAL_IMAGE_NAME with version '$APP_VERSION'"
docker buildx build --push \
--platform $BUILDX_PLATFORMS \
--build-arg appVersion=$APP_VERSION \
--tag $FINAL_IMAGE_NAME .
11 changes: 2 additions & 9 deletions .github/workflows/publish-dev.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,6 @@ on:
pull_request:
types: [labeled]
env:
GO111MODULE: on
# build only 2 platforms to speed up dev image build
# TODO create pre-release build label!
QEMU_PLATFORMS: arm64,arm
BUILDX_PLATFORMS: "linux/amd64,linux/arm64,linux/arm/v7"
IMAGE_REPO: flant/shell-operator-dev
Expand All @@ -14,19 +11,15 @@ jobs:
stub:
name: Empty job to prevent workflow fail
runs-on: ubuntu-latest
if: github.event_name == 'pull_request' && github.event.label.id != 1838515600 # not ':robot: build dev images' label
if: github.event_name == 'pull_request' && github.event.label.id != 1838515600 # not ':robot: build dev image: multiarch' label
steps:
- name: stub action
run: ": This job is used to prevent the workflow to fail when all other jobs are skipped."
# - name: dump label event
# run: cat $GITHUB_EVENT_PATH
# - name: dump envs
# run: export

unlabel:
name: Unlabel
runs-on: ubuntu-latest
if: github.event_name == 'pull_request' && github.event.label.id == 1838515600 # ':robot: build dev images' label
if: github.event_name == 'pull_request' && github.event.label.id == 1838515600 # ':robot: build dev image: multiarch' label
steps:
- uses: actions/github-script@v3
with:
Expand Down
1 change: 0 additions & 1 deletion .github/workflows/tests.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,6 @@ jobs:
run: |
go test \
-tags test \
-v \
./cmd/... ./pkg/... ./test/utils
prepare_build_dependencies:
Expand Down
225 changes: 225 additions & 0 deletions BINDING_VALIDATING.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,225 @@
# kubernetesValidating

This binding transforms a hook into a handler for ValidatingWebhookConfiguration. The Shell-operator creates ValidatingWebhookConfiguration, starts HTTPS server, and runs hooks to handle [AdmissionReview requests](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#request).

> Note: shell-operator use `admissionregistration.k8s.io/v1`, so Kubernetes 1.16+ is needed.
## Syntax

```yaml
configVersion: v1
onStartup: 10
kubernetes:
- name: myCrdObjects
...
kubernetesValidating:
- name: my-crd-validator.example.com
# include snapshots by binding names
includeSnapshotsFrom: ["myCrdObjects"]
# or use group name to include all snapshots in a group
group: "group name"
labelSelector: # equivalent of objectSelector
matchLabels:
label1: value1
...
namespace:
labelSelector: # equivalent of namespaceSelector
matchLabels:
label1: value1
...
matchExpressions:
- key: environment
operator: In
values: ["prod","staging"]
rules:
- apiVersions:
- v1
apiGroups:
- stable.example.com
resources:
- CronTab
operations:
- "*"
- operations: ["CREATE", "UPDATE"]
apiGroups: ["apps"]
apiVersions: ["v1", "v1beta1"]
resources: ["deployments", "replicasets"]
scope: "Namespaced"
failurePolicy: Ignore | Fail (default)
sideEffects: None (default) | NoneOnDryRun
timeoutSeconds: 2 (default is 10)
```
## Parameters
- `name` — a required parameter. It should be a domain with at least three segments separated by dots.

- `includeSnapshotsFrom` — an array of names of `kubernetes` bindings in a hook. When specified, a list of monitored objects from these bindings will be added to the binding context in the `snapshots` field.

- `group` — a key to include snapshots from a group of `schedule` and `kubernetes` bindings. See [grouping](#an-example-of-a-binding-context-with-group).

- `labelSelector` — [standard](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/#labelselector-v1-meta) selector of objects by labels (examples [of use](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels)). See [objectSelector](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#matching-requests-objectselector).

- `namespace.labelSelector` — this filter works like `labelSelector` but for namespaces. See [namespaceSelector](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#matching-requests-namespaceselector).

- `rules` — a required list of rules used to determine if a request to the Kubernetes API server should be sent to the hook. See [Rules](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#matching-requests-rules).

- `failurePolicy` — defines how errors from the hook are handled. See [Failure policy](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#failure-policy). Default is `Fail`.

- `sideEffects` — determine whether the hook is `dryRun`-aware. See [side effects](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#side-effects) documentation. Default is `None`.

- `timeoutSeconds` — a seconds API server should wait for a hook to respond before treating the call as a failure. See [timeouts](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#timeouts). Default is 10 (seconds).

As you can see, it is the close copy of a [Webhook configuration](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#webhook-configuration). Differences are:
- `objectSelector` is a `labelSelector` as in the `kubernetes` binding.
- `namespaceSelector` is a `namespace.labelSelector` as in the `kubernetes` binding.
- `clientConfig` is managed by the Shell-operator. You should provide a Service for the Shell-operator HTTPS endpoint. See example [204-validating-webhook](./examples/204-validating-webhook) for possible solution.
- `matchPolicy` is always "Equivalent". See [Matching requests: matchPolicy](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#matching-requests-matchpolicy).
- there are additional fields `group` and `includeSnapshotsFrom` to include snapshots in the binding context.

## Example

```
configVersion: v1
kubernetesValidating:
- name: private-repo-policy.example.com
rules:
- apiGroups: ["stable.example.com"]
apiVersions: ["v1"]
operations: ["CREATE"]
resources: ["crontabs"]
scope: "Namespaced"
```
The Shell-operator will execute hook with this configuration on every creation of CronTab object.
See example [204-validating-webhook](./examples/204-validating-webhook).
## Hook input and output
> Note that the `group` parameter is only for including snapshots. `kubernetesValidating` hook is never executed on `schedule` or `kubernetes` events with binding context with `"type":"Group"`.
The hook receives a binding context and should return response in `$VALIDATING_RESPONSE_PATH`.
$BINDING_CONTEXT_PATH file example:
```yaml
[{
# Name as defined in binding configuration.
"binding": "my-crd-validator.example.com",
# Validating to distinguish from other events.
"type": "Validating",
# Snapshots as defined by includeSnapshotsFrom or group.
"snapshots": { ... }
# AdmissionReview object.
"review": {
"apiVersion": "admission.k8s.io/v1",
"kind": "AdmissionReview",
"request": {
# Random uid uniquely identifying this admission call
"uid": "705ab4f5-6393-11e8-b7cc-42010a800002",
# Fully-qualified group/version/kind of the incoming object
"kind": {"group":"autoscaling","version":"v1","kind":"Scale"},
# Fully-qualified group/version/kind of the resource being modified
"resource": {"group":"apps","version":"v1","resource":"deployments"},
# subresource, if the request is to a subresource
"subResource": "scale",
# Fully-qualified group/version/kind of the incoming object in the original request to the API server.
# This only differs from `kind` if the webhook specified `matchPolicy: Equivalent` and the
# original request to the API server was converted to a version the webhook registered for.
"requestKind": {"group":"autoscaling","version":"v1","kind":"Scale"},
# Fully-qualified group/version/kind of the resource being modified in the original request to the API server.
# This only differs from `resource` if the webhook specified `matchPolicy: Equivalent` and the
# original request to the API server was converted to a version the webhook registered for.
"requestResource": {"group":"apps","version":"v1","resource":"deployments"},
# subresource, if the request is to a subresource
# This only differs from `subResource` if the webhook specified `matchPolicy: Equivalent` and the
# original request to the API server was converted to a version the webhook registered for.
"requestSubResource": "scale",
# Name of the resource being modified
"name": "my-deployment",
# Namespace of the resource being modified, if the resource is namespaced (or is a Namespace object)
"namespace": "my-namespace",
# operation can be CREATE, UPDATE, DELETE, or CONNECT
"operation": "UPDATE",
"userInfo": {
# Username of the authenticated user making the request to the API server
"username": "admin",
# UID of the authenticated user making the request to the API server
"uid": "014fbff9a07c",
# Group memberships of the authenticated user making the request to the API server
"groups": ["system:authenticated","my-admin-group"],
# Arbitrary extra info associated with the user making the request to the API server.
# This is populated by the API server authentication layer and should be included
# if any SubjectAccessReview checks are performed by the webhook.
"extra": {
"some-key":["some-value1", "some-value2"]
}
},
# object is the new object being admitted.
# It is null for DELETE operations.
"object": {"apiVersion":"autoscaling/v1","kind":"Scale",...},
# oldObject is the existing object.
# It is null for CREATE and CONNECT operations.
"oldObject": {"apiVersion":"autoscaling/v1","kind":"Scale",...},
# options contains the options for the operation being admitted, like meta.k8s.io/v1 CreateOptions, UpdateOptions, or DeleteOptions.
# It is null for CONNECT operations.
"options": {"apiVersion":"meta.k8s.io/v1","kind":"UpdateOptions",...},
# dryRun indicates the API request is running in dry run mode and will not be persisted.
# Webhooks with side effects should avoid actuating those side effects when dryRun is true.
# See http://k8s.io/docs/reference/using-api/api-concepts/#make-a-dry-run-request for more details.
"dryRun": false
}
}
}]
```

Response example:
```
cat <<EOF > $VALIDATING_RESPONSE_PATH
{"allowed": true}
EOF
```

Deny object creation and explain why:
```
cat <<EOF > $VALIDATING_RESPONSE_PATH
{"allowed": false, "message": "You cannot do this because it is Tuesday and your name starts with A"}
EOF
```

User will see an error message:

```
Error from server: admission webhook "policy.example.com" denied the request: You cannot do this because it is Tuesday and your name starts with A
```

Empty or invalid $VALIDATING_RESPONSE_PATH file is considered as `"allowed": false` with a short message about the problem and a more verbose error in the log.

## HTTP server and Kubernetes configuration

Shell-operator should create an HTTP endpoint with TLS support and register endpoints in the ValidatingWebhookConfiguration resource.

There should be a Service for shell-operator (see [Availability](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#availability)).

Command line options:

```
--validating-webhook-server-cert="/validating-certs/cert.crt"
A path to a server certificate for ValidatingWebhook. Can be set with $VALIDATING_WEBHOOK_SERVER_CERT.
--validating-webhook-server-key="/validating-certs/cert.key"
A path to a server private key for ValidatingWebhook. Can be set with $VALIDATING_WEBHOOK_SERVER_KEY.
--validating-webhook-ca="/validating-certs/ca.crt"
A path to a ca bundle for ValidatingWebhook. Can be set with $VALIDATING_WEBHOOK_CA.
--validating-webhook-client-ca=VALIDATING-WEBHOOK-CLIENT-CA ...
A path to a server certificate for ValidatingWebhook. Can be set with $VALIDATING_WEBHOOK_CLIENT_CA.
--validating-webhook-service-name=VALIDATING-WEBHOOK-SERVICE-NAME ...
A name of a service in front of a shell-operator. Can be set with $VALIDATING_WEBHOOK_SERVICE_NAME.
```
Loading

0 comments on commit 51b75bf

Please sign in to comment.