Skip to content

Latest commit

 

History

History
 
 

operator

README.md

StackRox Operator

Central Services and Secured Cluster Services operator.

Quickstart

All following commands should be run from this directory (operator/).

  1. Build and run operator locally. Note that this starts the operator without deploying it as a container in the cluster.
    It does not install any webhooks either. See Advanced usage for different ways of running operator.
make install run
  1. Create stackrox image pull secret in stackrox namespace.
    Helm charts use it by default to configure pods. If it does not exist, you'll need to specify ImagePullSecrets in custom resources.
make stackrox-image-pull-secret
  1. Create Central Custom Resource using the provided sample test sample.
kubectl -n stackrox delete persistentvolumeclaims stackrox-db

# TODO: switch back to user-facing samples in `config/samples/platform_v1alpha1_*.yaml`.
kubectl apply -n stackrox -f tests/common/central-cr.yaml
  1. Once Central services come online, create Secured Cluster using test sample.
# Get init-bundle secrets document from Central and save as secrets in the cluster
kubectl -n stackrox exec deploy/central -- \
  roxctl central init-bundles generate my-test-bundle --insecure-skip-tls-verify --password letmein --output-secrets - \
  | kubectl -n stackrox apply -f -

# Create Secured Cluster CR
kubectl apply -n stackrox -f tests/common/secured-cluster-cr.yaml
  1. Check status of the custom resource.
# For Central
kubectl get -n stackrox centrals.platform.stackrox.io
kubectl get -n stackrox centrals.platform.stackrox.io stackrox-central-services --output=json

# For Secured Cluster
kubectl get -n stackrox securedclusters.platform.stackrox.io
kubectl get -n stackrox securedclusters.platform.stackrox.io stackrox-secured-cluster-services --output=json
  1. Delete the custom resource.
# Central
kubectl delete centrals.platform.stackrox.io stackrox-central-services

# Secured Cluster
kubectl delete securedclusters.platform.stackrox.io stackrox-secured-cluster-services

Automated testing

This runs unit and integration tests using a minimum k8s control plane (just apiserver and etcd). Simply run:

$ make test

This runs end-to-end tests. Requires that your kubectl is configured to connect to a k8s cluster. Simply run:

$ make test-e2e

Secured Cluster Services

An example can be found in config/samples/platform_v1alpha1_securedcluster.yaml.

List all available commands/targets

To see help on other available Makefile targets, run

$ make help

Advanced usage

Launch the operator on the (local) cluster

While make install run can launch the operator, the operator is running outside the cluster and this approach may not be sufficient to test all aspects of it. An example are features that need to exercise any of the webhooks.

The recommended approach is the following.

  1. Make sure you have cert-manager installed. It takes care of the TLS aspects of the connection from k8s API server to the webhook server embedded in the manager binary.

  2. Build operator image

    $ make docker-build

  3. Make the image available for the cluster, this depends on k8s distribution you use.
    You don't need to do anything when using KIND.
    For minikube it could be done like this

    $ docker save stackrox/stackrox-operator:$(make tag) | ssh -o StrictHostKeyChecking=no -i $(minikube ssh-key) docker@$(minikube ip) docker load

  4. Install CRDs and deploy operator resources

    $ make install deploy

  5. Validate that the operator's pod has started successfully

    $ kubectl -n stackrox-operator-system describe pods

    Check logs

    $ kubectl -n stackrox-operator-system logs deploy/rhacs-operator-controller-manager manager -f

  6. Create CRs and have fun testing. Make sure you delete the CRs before you undeploy the operator resources. Otherwise, you'll need to clean up the operands yourself.

  7. When done

    $ make undeploy uninstall

Bundling

# Refresh bundle metadata. Make sure to check the diff and commit it.
$ make bundle
# Make sure that the operator is built and pushed
$ make docker-build docker-push
# Build and push bundle image
$ make bundle-build docker-push-bundle

Build and push everything as one-liner

$ make everything

Testing bundle with Scorecard

# Test locally-built bundle files
$ make bundle-test
# Test bundle image; the image must be pushed beforehand.
$ make bundle-test-image

Launch the operator on the cluster with OLM and the bundle

Note that unlike the make deploy route, deployment with OLM does not require cert-manager to be installed.

# 0. Get the operator-sdk program.
$ make operator-sdk

# 1. Install OLM.
$ bin/operator-sdk-1.9.0 olm install

# 2. Create a namespace for testing bundle.
$ kubectl create ns bundle-test

# 2. Create image pull secrets.
# If the inner magic does not work, just provide --docker-username and --docker-password with your DockerHub creds.
$ kubectl -n bundle-test create secret docker-registry my-opm-image-pull-secrets \
  --docker-server=https://index.docker.io/v1/ \
  --docker-email=ignored@email.com \
  $($(command -v docker-credential-osxkeychain || command -v docker-credential-secretservice) get <<<"docker.io" | jq -r '"--docker-username=\(.Username) --docker-password=\(.Secret)"')

# 3. Configure default service account to use these pull secrets.
$ kubectl -n bundle-test patch serviceaccount default -p '{"imagePullSecrets": [{"name": "my-opm-image-pull-secrets"}]}'

# 3. Build and push operator and bundle images.
# Use one-liner above.

# 4. Run bundle.
$ bin/operator-sdk-1.9.0 run bundle \
  docker.io/stackrox/stackrox-operator-bundle:v$(make --quiet tag) \
  --pull-secret-name my-opm-image-pull-secrets \
  --service-account default \
  --namespace bundle-test

# 5. Add image pull secrets to operator's ServiceAccount.
# Run it while the previous command executes otherwise it will fail.
# Note that serviceaccount might not exist for a few moments.
# Rerun this command until it succeeds.
# We hope that in OpenShift world things will be different and we will not have to do this.
$ kubectl -n bundle-test patch serviceaccount rhacs-operator-controller-manager -p '{"imagePullSecrets": [{"name": "my-opm-image-pull-secrets"}]}'
# You may need to bounce operator pods after this if they can't pull images for a while.
$ kubectl -n bundle-test delete pod -l app=rhacs-operator

# 6. operator-sdk run bundle command should complete successfully.
# If it does not, watch pod statuses and check pod logs.
$ kubectl -n bundle-test get pods
# ... and dive deep from there into the ones that are not healthy.

Once done, cleanup with

kubectl -n bundle-test delete clusterserviceversions.operators.coreos.com -l operators.coreos.com/rhacs-operator.bundle-test

kubectl -n bundle-test delete subscriptions.operators.coreos.com -l operators.coreos.com/rhacs-operator.bundle-test

kubectl -n bundle-test delete catalogsources.operators.coreos.com rhacs-operator-catalog

Also, you can blow everything away with

$ bin/operator-sdk-1.9.0 olm uninstall
$ kubectl delete ns bundle-test

Launch the Operator with OLM and Index

Note this assumes OLM is already in place.

So far only tested on an OpenShift cluster.

./hack/olm-operator-install.sh index-test $(make --quiet tag)

# undeploy

kubectl delete index-test

Extending the StackRox Custom Resource Definitions

Instructions and best practices on how to extend the StackRox CRDs is contained in the separate file EXTENDING_CRDS.md.