Live tests for AKS managed identity (Azure#9495)

danigm · Feb 10, 2020 · ad7e7a0 · ad7e7a0
1 parent c1d1262
commit ad7e7a0
Show file tree

Hide file tree

Showing 8 changed files with 356 additions and 1 deletion.
diff --git a/sdk/identity/azure-identity/tests/managed-identity-live/Dockerfile b/sdk/identity/azure-identity/tests/managed-identity-live/Dockerfile
@@ -6,7 +6,7 @@ ARG PYTHON_VERSION=2.7
 
 # docker can't tell when the repo has changed and will therefore cache this layer
 FROM alpine/git as repo
-RUN git clone https://github.com/chlowell/azure-sdk-for-python --single-branch --branch live-managed-id --depth 1 /azure-sdk-for-python
+RUN git clone https://github.com/Azure/azure-sdk-for-python --single-branch --branch master --depth 1 /azure-sdk-for-python
 
 
 FROM python:${PYTHON_VERSION}-slim

diff --git a/sdk/identity/azure-identity/tests/pod-identity/readme.md b/sdk/identity/azure-identity/tests/pod-identity/readme.md
@@ -0,0 +1,167 @@
+# Testing managed identity in Azure Kubernetes Service
+
+# prerequisite tools
+- Azure CLI
+  - https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest
+- Docker CLI
+  - https://hub.docker.com/search?q=&type=edition&offering=community
+- Helm 2.x (3.x doesn't handle CRDs properly at time of writing)
+  - https://github.com/helm/helm/releases
+
+
+# Azure resources
+This test requires instances of these Azure resources:
+- Azure Key Vault
+- Azure Managed Identity
+  - with secrets/set and secrets/delete permission for the Key Vault
+- Azure Container Registry
+- Azure Kubernetes Service
+  - RBAC requires additional configuration not provided here, so an RBAC-disabled cluster is preferable
+  - the cluster's service principal must have 'Managed Identity Operator' role over the managed identity
+  - must be able to pull from the Container Registry
+
+The rest of this section is a walkthrough of deploying these resources.
+
+### set environment variables to simplify copy-pasting
+- RESOURCE_GROUP
+  - name of an Azure resource group
+  - must be unique in the Azure subscription
+  - e.g. 'pod-identity-test'
+- AKS_NAME
+  - name of an Azure Kubernetes Service
+  - must be unique in the resource group
+  - e.g. 'pod-identity-test'
+- ACR_NAME
+  - name of an Azure Container Registry
+  - 5-50 alphanumeric characters
+  - must be globally unique
+- MANAGED_IDENTITY_NAME
+  - 3-128 alphanumeric characters
+  - must be unique in the resource group
+- KEY_VAULT_NAME
+  - 3-24 alphanumeric characters
+  - must begin with a letter
+  - must be globally unique
+
+### resource group
+```sh
+az group create -n $RESOURCE_GROUP --location westus2
+```
+
+### managed identity
+Create the managed identity:
+```sh
+az identity create -g $RESOURCE_GROUP -n $MANAGED_IDENTITY_NAME
+```
+
+Save its `clientId`, `id` (ARM URI), and `principalId` (object ID) for later:
+```sh
+export MANAGED_IDENTITY_CLIENT_ID=$(az identity show -g $RESOURCE_GROUP -n $MANAGED_IDENTITY_NAME --query clientId -o tsv) \
+       MANAGED_IDENTITY_ID=$(az identity show -g $RESOURCE_GROUP -n $MANAGED_IDENTITY_NAME --query id -o tsv) \
+       MANAGED_IDENTITY_PRINCIPAL_ID=$(az identity show -g $RESOURCE_GROUP -n $MANAGED_IDENTITY_NAME --query principalId -o tsv)
+```
+
+### Key Vault
+Create the Vault:
+```sh
+az keyvault create -g $RESOURCE_GROUP -n $KEY_VAULT_NAME --sku standard
+```
+
+Add an access policy for the managed identity:
+```sh
+az keyvault set-policy -n $KEY_VAULT_NAME --object-id $MANAGED_IDENTITY_PRINCIPAL_ID --secret-permissions set delete
+```
+
+### container registry
+```sh
+az acr create -g $RESOURCE_GROUP -n $ACR_NAME --admin-enabled --sku basic
+```
+
+### Kubernetes
+Deploy the cluster (this will take several minutes):
+```sh
+az aks create -g $RESOURCE_GROUP -n $AKS_NAME --generate-ssh-keys --node-count 1 --disable-rbac --attach-acr $ACR_NAME
+```
+
+Grant the cluster's service principal permission to use the managed identity:
+```sh
+az role assignment create --role "Managed Identity Operator" \
+  --assignee $(az aks show -g $RESOURCE_GROUP -n $AKS_NAME --query servicePrincipalProfile.clientId -o tsv) \
+  --scope $MANAGED_IDENTITY_ID
+```
+
+
+# build images
+The test application must be packaged as a Docker image before deployment.
+Test runs must include Python 2 and 3, so two images are required.
+
+### authenticate to ACR
+```sh
+az acr login -n $ACR_NAME
+```
+
+### acquire the test code
+```sh
+git clone https://github.com/Azure/azure-sdk-for-python/ --branch master --single-branch --depth 1
+```
+
+The rest of this section assumes this working directory:
+```sh
+cd azure-sdk-for-python/sdk/identity/azure-identity/tests
+```
+
+### build images and push them to the container registry
+Set environment variables:
+```sh
+export REPOSITORY=$ACR_NAME.azurecr.io IMAGE_NAME=test-pod-identity PYTHON_VERSION=2.7
+```
+
+Build an image:
+```sh
+docker build --no-cache --build-arg PYTHON_VERSION=$PYTHON_VERSION -t $REPOSITORY/$IMAGE_NAME:$PYTHON_VERSION ./managed-identity-live
+```
+
+Push it to ACR:
+```sh
+docker push $REPOSITORY/$IMAGE_NAME:$PYTHON_VERSION
+```
+
+Then set `PYTHON_VERSION` to the latest 3.x (3.8 at time of writing) and run the
+above `docker build` and `docker push` commands again. (It's safe--and faster--
+to omit `--no-cache` from `docker build` the second time.)
+
+
+# run the test
+
+### install kubectl
+```sh
+az aks install-cli
+```
+
+### authenticate kubectl and helm
+```sh
+az aks get-credentials -g $RESOURCE_GROUP -n $AKS_NAME
+```
+
+### install tiller
+```sh
+helm init --wait
+```
+
+### run the test script
+Twice. Once with `PYTHON_VERSION=2.7`, once with `PYTHON_VERSION=3.x`
+(replacing x with the latest Python 3 minor version):
+```sh
+python ./pod-identity/run-test.py \
+ --client-id $MANAGED_IDENTITY_CLIENT_ID \
+ --resource-id $MANAGED_IDENTITY_ID \
+ --vault-url https://$KEY_VAULT_NAME.vault.azure.net \
+ --repository $REPOSITORY \
+ --image-name $IMAGE_NAME \
+ --image-tag $PYTHON_VERSION
+```
+
+### delete Azure resources
+```sh
+az group delete -n $RESOURCE_GROUP -y --no-wait
+```
diff --git a/sdk/identity/azure-identity/tests/pod-identity/run-test.py b/sdk/identity/azure-identity/tests/pod-identity/run-test.py
@@ -0,0 +1,94 @@
+# ------------------------------------
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT License.
+# ------------------------------------
+"""Deploys the test app and prints its output."""
+
+import argparse
+import os
+import subprocess
+import sys
+import time
+
+JOB_NAME = "test"
+HELM_APP_NAME = "test"
+
+parser = argparse.ArgumentParser()
+parser.add_argument("--client-id", required=True, help="managed identity's client ID")
+parser.add_argument("--resource-id", required=True, help="managed identity's ARM ID")
+parser.add_argument("--vault-url", required=True, help="URL of a vault whose secrets the managed identity may manage")
+parser.add_argument("--verbose", "-v", action="store_true", help="print all executed commands and their output")
+
+image_options = parser.add_argument_group("image", "image options")
+image_options.add_argument("--repository", required=True, help="repository holding the test image")
+image_options.add_argument("--image-name", required=True, help="name of the test image")
+image_options.add_argument("--image-tag", required=True, help="test image tag")
+
+args = parser.parse_args()
+
+
+def run_command(command, exit_on_failure=True):
+    try:
+        if args.verbose:
+            print(" ".join(command))
+        result = subprocess.check_output(command, stderr=subprocess.STDOUT).decode("utf-8").strip("'")
+        if args.verbose:
+            print(result)
+        return result
+    except subprocess.CalledProcessError as ex:
+        result = ex.output.decode("utf-8").strip()
+        if exit_on_failure:
+            print(result)
+            sys.exit(1)
+        return result
+
+
+# install the chart
+helm_install = [
+    "helm",
+    "install",
+    os.path.join(os.path.dirname(__file__), "test-pod-identity"),
+    "-n",
+    HELM_APP_NAME,
+    "--set",
+    "aad-pod-identity.azureIdentity.resourceID={},aad-pod-identity.azureIdentity.clientID={}".format(
+        args.resource_id, args.client_id
+    ),
+    "--set",
+    "vaultUrl=" + args.vault_url,
+    "--set",
+    "image.repository={},image.name={},image.tag={}".format(args.repository, args.image_name, args.image_tag),
+]
+run_command(helm_install)
+
+# get the name of the test pod
+pod_name = run_command(
+    ["kubectl", "get", "pods", "--selector=job-name=" + JOB_NAME, "--output=jsonpath='{.items[*].metadata.name}'"]
+)
+
+logs = ""
+
+# poll the number of active pods to determine when the test has finished
+count_active_pods = ["kubectl", "get", "job", JOB_NAME, "--output=jsonpath='{.status.active}'"]
+for _ in range(10):
+    # kubectl will return '' when there are no active pods
+    active_pods = run_command(count_active_pods)
+    logs = run_command(["kubectl", "logs", "-f", pod_name], exit_on_failure=False)
+    if not active_pods:
+        break
+    time.sleep(30)
+
+# output logs from the most recent run
+print(logs)
+
+# uninstall the chart
+run_command(["helm", "del", "--purge", HELM_APP_NAME])
+
+# delete CRDs because Helm didn't
+pod_identity_CRDs = [
+    "azureassignedidentities.aadpodidentity.k8s.io",
+    "azureidentities.aadpodidentity.k8s.io",
+    "azureidentitybindings.aadpodidentity.k8s.io",
+    "azurepodidentityexceptions.aadpodidentity.k8s.io",
+]
+run_command(["kubectl", "delete", "crd"] + pod_identity_CRDs)
diff --git a/sdk/identity/azure-identity/tests/pod-identity/test-pod-identity/.helmignore b/sdk/identity/azure-identity/tests/pod-identity/test-pod-identity/.helmignore
@@ -0,0 +1,22 @@
+# Patterns to ignore when building packages.
+# This supports shell glob matching, relative path matching, and
+# negation (prefixed with !). Only one pattern per line.
+.DS_Store
+# Common VCS dirs
+.git/
+.gitignore
+.bzr/
+.bzrignore
+.hg/
+.hgignore
+.svn/
+# Common backup files
+*.swp
+*.bak
+*.tmp
+*~
+# Various IDEs
+.project
+.idea/
+*.tmproj
+.vscode/
diff --git a/sdk/identity/azure-identity/tests/pod-identity/test-pod-identity/Chart.yaml b/sdk/identity/azure-identity/tests/pod-identity/test-pod-identity/Chart.yaml
@@ -0,0 +1,16 @@
+# ------------------------------------
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT License.
+# ------------------------------------
+apiVersion: v1
+name: test-pod-identity
+description: Helm chart for deploying a pod identity test app to Kubernetes
+
+type: application
+
+version: 0.1.0
+
+dependencies:
+  - name: aad-pod-identity
+    repository: https://raw.githubusercontent.com/Azure/aad-pod-identity/master/charts
+    version: 1.5.5
diff --git a/...ity/azure-identity/tests/pod-identity/test-pod-identity/charts/aad-pod-identity-1.5.5.tgz b/...ity/azure-identity/tests/pod-identity/test-pod-identity/charts/aad-pod-identity-1.5.5.tgz
diff --git a/sdk/identity/azure-identity/tests/pod-identity/test-pod-identity/templates/job.yaml b/sdk/identity/azure-identity/tests/pod-identity/test-pod-identity/templates/job.yaml
@@ -0,0 +1,34 @@
+# ------------------------------------
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT License.
+# ------------------------------------
+
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: test
+  labels:
+    app: pod-identity-test
+spec:
+  backoffLimit: 12  # give up after this many attempts
+  ttlSecondsAfterFinished: 3600  # delete the job and its sub-resources after this many seconds
+  template:
+    metadata:
+        labels:
+          app: pod-identity-test
+          aadpodidbinding: pod-identity-test
+    spec:
+      restartPolicy: OnFailure  # ensure we have only one pod, whose logs reflect the last test run
+      initContainers:
+      - name: wait-for-imds  # this container exits successfully when the IMDS endpoint returns 200 when asked for a
+        image: busybox:1.31  # Key Vault token, guaranteeing IMDS is configured and ready before the test runs
+        command: ['sh', '-c', 'wget "http://169.254.169.254/metadata/identity/oauth2/token?resource=https://vault.azure.net&api-version=2018-02-01" --header "Metadata: true" -S --spider -T 6']
+      containers:
+      - name: test-pod-identity
+        image: "{{ .Values.image.repository }}/{{ .Values.image.name }}:{{ .Values.image.tag }}"
+        imagePullPolicy: Always
+        env:
+        - name: AZURE_IDENTITY_TEST_VAULT_URL
+          value: "{{ .Values.vaultUrl }}"
+        - name: AZURE_IDENTITY_TEST_MANAGED_IDENTITY_CLIENT_ID
+          value: {{ index .Values "aad-pod-identity" "azureIdentity" "clientID" | quote }}
diff --git a/sdk/identity/azure-identity/tests/pod-identity/test-pod-identity/values.yaml b/sdk/identity/azure-identity/tests/pod-identity/test-pod-identity/values.yaml
@@ -0,0 +1,22 @@
+# ------------------------------------
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT License.
+# ------------------------------------
+
+# Default values for test-pod-identity
+
+image:
+  repository: ""
+  name: ""
+  tag: ""
+  pullPolicy: Always
+
+vaultUrl: ""
+
+# override values for aad-pod-identity
+aad-pod-identity:
+  azureIdentityBinding:
+    name: "pod-identity-test-binding"
+    selector: "pod-identity-test"
+  azureIdentity:
+    enabled: true