A tool to keep your Kubernetes secrets in sync with Vault

Lots of companies use Vault as their secrets store backend for multiple kind of secrets and different purposes. Kubernetes brings a nice secrets API, but it means that you have two different sources of truth for your secrets.

secrets-manager tries to solve this, by reading secrets from Vault and comparing them to Kubernetes secrets, creating and updating them as you do it in Vault.

• cert-manager. cert-manager solves a different issue, automation around issuing and renewing certificates. It integrates with Let's Encrypt and Vault (using the pki backend) being those the certificates issuer. While this is really powerful and really a tool which is fully compatible with secrets-manager, it does not really sync a secret from a secret backend. secrets-manager is a more generic tool where you can sync certificates or any kind of secret from the source of truth of your secrets to Kubernetes secrets.

• vault-operator. This manages vault clusters in Kubernetes, so it is a completely different tool.

• vault-crd. This is the tool that really inspired secrets-manager. We opened this issue asking for token renewal or other login mechanism. While the author is very responsive answering, we could not wait for an implementation and since we were are more familiar with Go than Java we decided to write secrets-manager. We are very thankful to the author of vault-crd, since it has been really inspiring. Some differences:

  • vault-crd only supports Hashicorp Vault as its secrets manager, while secrets-manager has been designed to support other backends (we only support Vault for now, though).
  • vault-crd supports KV1, KV2 and PKI secret engines. secrets-manager supports KV1 and KV2. It is on our roadmap to support more secret engines.

secrets-manager will login to Vault using AppRole credentials and it will start a reconciliation loop watching for changes in SecretsDefinition objects. In background it will run two main operations:

• If Vault token is close to expire and if that's the case, renewing it. If it can't renew, it will try to re-login.
• It will re-queue SecretsDefinition events and in every event loop it will verify if the current Kubernetes secret it is in the desired state by comparing it with the data in Vault and creating/updating them accordingly

secrets-manager now uses Custom Resource Definitions to extend Kubernetes APIs with a new SecretDefinition object that it will watch.

To install the CRD in your cluster: kubectl apply -f crd.yaml

• name: This will be the name of the secret created in Kubernetes.
• type: Kubernetes secret type. One of kubernetes.io/tls, Opaque.
• keysMap: This will contain the Kubernetes secret data keys as a map of datasources. Each datasource will contain the way to access the secret in the secret backend source of truth, via a path and a key. And optional encoding key can be provided if your secrets are codified in base64. The absence of encoding or encoding: text means no encoding.

NOTE: We let the user all the responsibility to set the whole Vault path. So it is important to know which path a secret engine needs to be set. For instance, with the KV version 1 all secrets are stored in secret/ whereas with the KV version 2, all secrets go under secret/data/

An example of a secretdefinition object

$ cat > secretdefinition-sample.yaml <<EOF
---
apiVersion: secrets-manager.tuenti.io/v1alpha1
kind: SecretDefinition
metadata:
  name: secretdefinition-sample
spec:
  # Add fields here
  name: supersecretnew
  keysMap:
    decoded:
      path: secret/data/pathtosecret1
      encoding: base64
      key: value
    raw:
      path: secret/data/pathtosecret1
      key: value

EOF

To deploy it just run kubectl apply -f secretdefinition-sample.yaml

Secrets Manager can be run in one of 2 ways:

• Global secrets management in all namespaces for the whole of a Kuberentes cluster
• Manage specific namespaces

In order for Secrets Manager to act as a manager for all Namespaces it requires a ClusterRole that enables it to manage all secrets and secretdefinitions in the entire Kubernetes cluster as in the config/rbac/role.yaml and config/rbac/rolebinding.yaml examples.

Alternatively if you use the watch-namespaces argument to limit secretdefinition monitoring to sepcific namespaces then you can just give the serviceAccount that secrets-manager is running as a standard role and a rolebinding in each of the namespaces that you want it to manage as shown in the config/rbac/secrets_manager_role.yaml and config/rbac/secrets_manager_role_binding.yaml examples. Alternatively you can still use a cluster role if you so wish.

To be able to interact with secretdefinition resources using the standard admin, edit and view Kubernetes native roles, you need to create the following ClusterRole aggregations:

• config/rbac/secretdefinitions_admin_clusterrole_aggregation.yaml
• config/rbac/secretdefinitions_view_clusterrole_aggregation.yaml

More information about aggregated ClusterRoles can be found at kubernetes.io > Using RBAC Authorization > Aggregated ClusterRoles.

secrets-manager exposes the following Prometheus metrics at http://$cfg.listen-addr/metrics:

We do recommend you use policies to make sure you grant secrets-manager only to those secrets you need available in your Kubernetes cluster. An example of simple policy could be:

path "secret/data/my-k8s-cluster/*" {
  capabilities = ["read"]
}

To create this policy:

$ cat > my-policy.hcl <<EOF
path "secret/data/my-k8s-cluster/*" {
  capabilities = ["read"]
}
EOF

$ cat my-policy.hcl | vault policy write my-policy -

Vault tokens will be renewed by secrets-manager if the ttl is lower than vault.max-token-ttl and the token is renewable. But as per Vault's documentation, regular tokens will have their own max TTL that it's calculated on every renewal, so that a token will eventually expire. This can be ok for your use case, but for others a periodic token could be much more convinient. In the case of a periodic token, the period will invalidate the vault.renew-ttl-increment option.

Vault token as a login mechanism has been deprecated in favor of the AppRole authentication method for secrets-manager. secrets-manager will still renew the token obtained after login in, but will make secrets-manager more resilient in case of a token has expired due to network issues, Vault sealed, etc.

So instead of expecting a token, secrets-manager expects a role_id and a secret_id to connect to Vault.

To create a role with a permanent secret_id attached to a policy:

$ vault write auth/approle/role/secrets-manager policies=my-policy secret_id_num_uses=0 secret_id_ttl=0

To get a secret_id:

$ vault write -force auth/approle/role/secrets-manager/secret-id

To get the role_id:

$ vault read auth/approle/role/secrets-manager/role-id

In addition to appRole, secrets-manager can authenticate to Vault using its own Kubernetes serviceAccount. Follow the Vault Kubernetes auth guide to enable it and configure it.

Example:

$ cat > secrets-manager-role.json <<EOF
{
  "bound_service_account_names": [ "secrets-manager" ],
  "bound_service_account_namespaces": [ "my-namesapace" ],
  "policies": [ "my-policy" ],
  "max_ttl": 3600
}
EOF

$ vault write auth/kubernetes/role/secrets-manager @secrets-manager-role.json

If you haven't still deployed an Azure KeyVault server, you can do it with Azure CLI:

$ az keyvault create --location <location> --name <keyvault_name> --resource-group <resource_group>

Secrets manager currently supports the following authentication methods for Azure:

• Azure Managed Identity. This is the preferred method if secrets manager is running in an Azure VM or Azure Kubernetes Service (AKS) cluster. For further information about how to use managed identities in Azure Kubernetes Service (AKS) see documentation.
• Azure Service Principal. This method can be used if secrets manager runs outside of Azure service, although it requires more configuration steps.

secrets-manager uses Azure Service Principal to authenticate against Azure KeyVault API. It's recommended to use an isolated Service Principal to limit its access to the least required resources:

$ az ad sp create-for-rbac --name "<service_priciple_name>" --role Contributor --scopes /subscriptions/{SubID}/resourceGroups/{ResourceGroup}

This command will output a JSON object like the following:

{
  "appId": "<AppID>", // ClientId
  "displayName": "<ServicePrincipleName>",
  "name": "http://<ServicePrincipleName>",
  "password": "<Password>", // ClientSecret
  "tenant": "<TenantId>"
}

The fields needed by secrets-manager to authenticate are appId (azure-kv.client-id), password (azure-kv.client-secret) and tenant (azure-kv.tenant-id).

Once the Service Principal is created, add permission to access Azure KeyVault's secrets with:

$ az keyvault set-policy --name <keyvault_name> --spn <appId> --secret-permissions get list set delete

Right now versioning it's a manually task. Depending on the kind of the update we would apply a major, minor or patch update given that we follow semantic versioning.

Before building release images, we should run one of the following commands:

• make update-major-version
• make update-minor-version
• make update-patch-version

secrets-manager has been designed to be deployed in Kubernetes, you will find a full deployment example in the config/samples folder.

secrets-manager is developed and maintained by Tuenti Technologies S.L.

You can follow Tuenti engineering team on Twitter @tuentieng.

secrets-manager is available under the Apache License, Version 2.0. See LICENSE file for more info.