• Author(s): Dako Dakov (ddakov@gmail.com)
• Status: implemented

• Fill out this file as best you can. There are instructions as HTML comments. At minimum, you should fill in the "Summary" and "Motivation" sections.
• Create a PR for this VEP.
• Merge early and iterate. Avoid getting hung up on specific details and instead aim to get the goals of the VEP clarified and merged quickly. The best way to do this is to just start with the high-level sections and fill out details incrementally in subsequent PRs.

• Summary
• Glossary
• Motivation
• Requirements and goals
• High-level design
• API Design
• Detailed design
• Implementation stories
• Alternatives

Provide integration with Hashicorp Vault for secure storage of secrets. Control Service operators will be able to optionally configure a vault instance to be used for storing data job properties which are marked as secret.

CS or VDK-CS - The Versatile Data Kit Control Service Hashicorp Vault or Vault - A tool for secrets management, encryption as a service, and privileged access management Secret/Secrets - sensitive data which needs to be available to data jobs - including but no limited to: secrets/ passwords/credentials/tokens/data

• Some data jobs need to use secrets in order to connect to third party systems by providing user credentials or tokens - currently these secrets are stored in data jobs properties, which in turn are stored in plain text in the configured Control Service Database. This enhancement will allow users to securely store secrets/passwords/credentials/tokens/data into a vault.
• Compliance: Storing sensitive information such as credentials, tokens, or API keys in plaintext may lead to non-compliance with certain industry standards or regulations, such as GDPR, HIPAA. Without it, VDK would be hard to be certified to store or process restricted type of data.
• Risk reduction: Storing secrets in plaintext increases the risk of unauthorized access and potential data breaches. This is severely mitigated by services like Vault .

Goals:

• provide the ability to optionally configure a Vault instance in the Control Service for storing secrets
• change the APIs and SDK so that users can set and retrieve secrets and data jobs can use them during runtime

Non-Goals:

• automatic provisioning of a Vault instance as part of the VDK-CS installation
• automatic migration of existing secrets to Vault

In order to implement the proposed change, we should make the following changes - highlighted in the diagram below:

• provide the ability to optionally configure a Vault instance in the Control Service for storing secrets
• change the CS to store properties marked as secret into the configured Vault
• enhance vdk cli to allow for users to store secrets for their data jobs
• during runtime, data jobs can retrieve/store/update the secrets

Currently, data job properties are stored and read in bulk. We are going to introduce similar APIs for storing and reading secrets.

Introduce new API methods for storing and retrieving secrets under: /data-jobs/for-team/{team_name}/jobs/{job_name}/deployments/{deployment_id}/secrets

...
    Secrets:
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/DataJobSecrets'
      description: Data Job properties


...
    DataJobSecrets:
      description: Secrets of a Data Job
      type: object
      additionalProperties:
        type: object
      example:
        redshift-user: foo
        redshift-password: bar
...

  '/data-jobs/for-team/{team_name}/jobs/{job_name}/deployments/{deployment_id}/secrets':
    summary: |
      Data Job Secrets API.
    get:
      tags:
        - Data Jobs Secrets
      responses:
        '200':
          $ref: '#/components/responses/Secrets'
      operationId: dataJobSecretsRead
      summary: Get Data Job secrets.
      description: <i>(Introduced in v1.0)</i>
    put:
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/DataJobSecrets'
        required: true
      tags:
        - Data Jobs Secrets
      responses:
        '201':
          description: Created successfully.
        '204':
          description: Updated successfully.
      operationId: dataJobSecretsUpdate
      summary: Update Data Job secrets.
      description: <i>(Introduced in v1.0)</i>
    parameters:
      -  name: team_name
         description: Team Name
         schema:
           type: string
         in: path
         required: true
      - name: job_name
        description: Data Job Name.
        schema:
          type: string
        in: path
        required: true
      - name: deployment_id
        description: Data job deployment id.
        schema:
          $ref: '#/components/schemas/DataJobDeploymentId'
        in: path
        required: true

When storing secrets, the request body is expected to provide a JSON object, containing all secrets:

{
  "secret1": "secret_value1",
  "secret2": "secret_value2",
  "secret3": "secret_value3"
}

We are going to introduce Secrets API similar to the existing Properties API in the JobInput class which allow data jobs to get and set secrets:

class ISecrets:
    """
    Allows for Data Job to store and retrieve secrets.
    Secrets are a solution for securely storing API keys, tokens and passwords necessary to connect to different systems
    """

    @abstractmethod
    def get_secret(self, name: str, default_value: Any = None) -> str:
        pass

    @abstractmethod
    def get_all_secrets(self) -> dict:
        pass

    @abstractmethod
    def set_all_secrets(self, properties: dict):
        pass

And here are examples of the usage:

Get a single secret:

def run(job_input):
   my_secret_token = job_input.get_secret("my_secret_key")

Get all secrets:

def run(job_input):
   secrets = job_input.get_all_secrets()
   aws_token = secrets['MY_AWS_TOKEN']
   authenticate(aws_token)

Set a secrets:

def run(job_input):
   secrets = job_input.get_all_secrets()
   secrets['MY_TOKEN'] = get_new_token()
   job_input.set_all_secrets(secrets)

We are going to introduce a new set of commands for secrets, similar to "vdk properties" where users will be prompted to set to enter each password.

• Will prompt for the value so it's not printed on the screen vdk secrets --set "my-password"

• Return the secret value associated with the given key "my-password" vdk secrets --get "my-password"

• List all secret keys vdk secrets --list

• List all secret keys and values vdk secrets --list --show-secrets

• Delete a secret vdk secrets --delete "my-password"

• Delete all job secrets vdk secrets --delete-all-job-properties

• Overwrite all job secrets by passing a JSON file vdk secrets --overwrite-all-job-secrets

Changes to the properties cli command:

• remove the --set-secret option
• change the explanation of the command from "store credentials securely" to "store credentials"

We are going to enhance the VDK-CS configuration with an optional Spring Vault Configuration.

# Hashicorp Vault Integration settings
# When disabled/not configured the Secrets functionality won't work
featureflag.vault.integration.enabled=true
vdk.vault.uri=https://localhost:8200/v1/
vdk.vault.approle.roleid=z2d59142-9b53-d163-1ae9-d6286d3dfe22
vdk.vault.approle.secretid=c336f9e9-1555-8c29-9a7f-283608d06fbd
datajobs.vault.size.limit.bytes=1048576

The feature flag allows users who are not interested in using a secret storage, to simply disable this feature.

The VDK Control Service is going to authenticate using Vault's AppRole mechanism. THe Service Operator should setup the App Role authentication based on their needs in Vault, and provide the Vault URI the App Role's Role ID and Secret ID.

We are going to introduce a new Secrets Service class, similar to the existing Properties Service which handles requests for the Secrets functionality. We are going to implement changes in VDK-CS API as outlined API Design section.

The secrets for a given data job will be stored under "secret//" entry in Vault in JSON object format. So the secrets for a data job "my-ingestion-job" belonging to the "ingestion" team will be stored under "secret/ingestion/my-ingestion-job".

The maximum size of a secret entry in Vault is 1MiB. Upon calling the store secret method, the VDK-CS will check the size of the secret object and will return an error if the secret is above the limit. The VDK SDK will interpret this error as a user error.

Once the changes in the API have been introduced, we are going to implement the changes in the VDK SDK, e.g. introduce the secrets related methods in classes, similar to the existing job_properties built-in plugin as outlined API Design section.

As soon as the changes in the VDK SDK are completed, we can introduce the secrets related commands in the VDK SDK.

To improve data job performance, the secrets will be loaded and cached at the start of the SDK run job - the same functionality we currently have for properties.

During the initial architecture of this feature we had a discussion on whether the secrets should be part of the existing properties api/interface. We also reached out to several people whom we consider power users of VDK and everyone agreed that the secrets should be part of a separate interface as this would eliminate ambiguity and provide clear separation between the two types of data.

Also storing secrets separately would overall decrease the number of updates of secrets which is generally undesirable due to Vault's characteristics where frequent updates of secrets could lead to high latency and reduced performance.

We are not going to start storing all properties in vault as secrets as Vault is not designed to handle frequent updates of the entries stored within.

Our general guidance for properties is that they should be used to store state, e.g. last time stamp or last record of processed data which could change frequently. We have observed data jobs where users update the properties quite often, multiple times per single data job run, which is not suitable for secrets stored in Vault.

To that end we are going to keep the two functionalities separate.