[FEATURE] Support volume encryption for (encrypted) backing image volumes #7051

innobead · 2023-11-06T14:10:21Z

Is your feature request related to a problem? Please describe (👍 if you like this request)

Currently, Longhorn supports non-backing image volume encryption, and it is impossible to enable volume encryption for an unencrypted backing image. The goal here is to support volume encryption regardless of whether the volume is based on a backing image or not, with the potential condition being that the backing image is also encrypted using the same crypto key.

ref: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSEncryption.html#how-ebs-encryption-works

Encrypt unencrypted resources
You cannot directly encrypt existing unencrypted volumes or snapshots. However, you can create encrypted volumes or snapshots from unencrypted volumes or snapshots. If you enable encryption by default, Amazon EBS automatically encrypts new volumes and snapshots using your default KMS key for EBS encryption.

ref: https://gitlab.com/cryptsetup/cryptsetup/-/wikis/FrequentlyAskedQuestions

2.5 Can I encrypt an existing, non-empty partition to use LUKS?

There is no converter, and it is not really needed. The way to do this is to make a backup of the device in question, securely wipe the device (as LUKS device initialization does not clear away old data), do a luksFormat, optionally overwrite the encrypted device, create a new filesystem and restore your backup on the now encrypted device. Also refer to sections "Security Aspects" and "Backup and Data Recovery".

Describe the solution you'd like

When creating a first snapshot from the backing image, should be encrypted first.

Describe alternatives you've considered

Additional context

ChanYiLin · 2024-05-08T10:46:16Z

Experiments

Create Encrypted Volume with BackingImage - FAILED

Create StorageClass with BackingImage and Encryption Info

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: longhorn-crypto-global
provisioner: driver.longhorn.io
allowVolumeExpansion: true
parameters:
  numberOfReplicas: "3"
  staleReplicaTimeout: "2880" # 48 hours in minutes
  fromBackup: ""
  encrypted: "true"
  backingImage: "parrot"
  backingImageDataSourceType: "download"
  backingImageDataSourceParameters: '{"url": "https://longhorn-backing-image.s3-us-west-1.amazonaws.com/parrot.raw"}'
  csi.storage.k8s.io/provisioner-secret-name: "longhorn-crypto"
  csi.storage.k8s.io/provisioner-secret-namespace: "longhorn-system"
  csi.storage.k8s.io/node-publish-secret-name: "longhorn-crypto"
  csi.storage.k8s.io/node-publish-secret-namespace: "longhorn-system"
  csi.storage.k8s.io/node-stage-secret-name: "longhorn-crypto"
  csi.storage.k8s.io/node-stage-secret-namespace: "longhorn-system"

Create PVC from the StorageClass

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: longhorn-backing-image-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn-crypto-global
  resources:
    requests:
      storage: 5Gi

Create Pod using the Volume

apiVersion: v1
kind: Pod
metadata:
  name: longhorn-simple-pod
  namespace: default
spec:
  nodeSelector:
    kubernetes.io/hostname: kworker1
  restartPolicy: Always
  containers:
    - name: ubuntu
      image: ubuntu:22.04
      imagePullPolicy: IfNotPresent
      command: [ "/bin/bash", "-c", "--" ]
      args: [ "while true; do sleep 30; done;" ]
      volumeMounts:
        - name: vol
          mountPath: /data
      ports:
        - containerPort: 80
  volumes:
    - name: vol
      persistentVolumeClaim:
        claimName: longhorn-backing-image-pvc

Failed
Got error message

Warning  FailedMount             14s (x7 over 46s)  kubelet                  MountVolume.MountDevice failed for volume "pvc-be8862f6-7b95-42dc-8067-0d3985cb757c" : 
rpc error: code = InvalidArgument desc = unsupported disk encryption format unknown data, probably partitions

ChanYiLin · 2024-05-08T10:53:36Z

Experiments

Create Encrypted Volume with Encrypted BackingImage - SUCCEEDED

Create a Encrypted Volume VolumeA first
Write some data
Export the Volume to BackingImage BackingImageA

Create StorageClass with BackingImageA and Encryption Info

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: longhorn-crypto-global-export
provisioner: driver.longhorn.io
allowVolumeExpansion: true
parameters:
  numberOfReplicas: "3"
  staleReplicaTimeout: "2880" # 48 hours in minutes
  fromBackup: ""
  encrypted: "true"
  backingImage: "parrot-encrypt"
  backingImageDataSourceType: "export-from-volume"
  csi.storage.k8s.io/provisioner-secret-name: "longhorn-crypto"
  csi.storage.k8s.io/provisioner-secret-namespace: "longhorn-system"
  csi.storage.k8s.io/node-publish-secret-name: "longhorn-crypto"
  csi.storage.k8s.io/node-publish-secret-namespace: "longhorn-system"
  csi.storage.k8s.io/node-stage-secret-name: "longhorn-crypto"
  csi.storage.k8s.io/node-stage-secret-namespace: "longhorn-system"

Create the PVC from the StorageClass
Create Pod using the PVC
Successfully Mounted

ChanYiLin · 2024-05-08T17:10:26Z

Other Reference

AWS

AWS supports following three scenarios: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIEncryption.html

Encrypt a volume during launch
- In this example, an AMI backed by an unencrypted snapshot is used to launch an EC2 instance with an encrypted EBS volume.
Re-encrypt a volume during launch
- In this example, an AMI backed by an encrypted snapshot is used to launch an EC2 instance with an EBS volume encrypted by a new KMS key.
Change encryption state of multiple volumes during launch
- In this more complex example, an AMI backed by multiple snapshots (each with its own encryption state) is used to launch an EC2 instance with a newly encrypted volume and a re-encrypted volume.

Ceph

Ceph supports encrypt the rdb image: https://docs.ceph.com/en/quincy/rbd/rbd-encryption/

ChanYiLin · 2024-05-10T16:13:07Z

Experiments

Manually Encrypt a Non-encrypted Raw Backing Image - SUCCEEDED

Create a BackingImage with testing raw image
- https://longhorn-backing-image.s3-us-west-1.amazonaws.com/parrot.raw
- 32MB
Login to the host and cd to /var/lib/longhorn/backing-images/parrot-cd8eb5b3/
Truncate an empty image
- With the size = image.size + 16MB (dm-crypt metadata(16MB))
- $ truncate -s 48M backing-encrypted
Setup the loop device
- $ losetup /dev/loop7 backing-encrypted
Format the device with cryptsetup
- $ echo -n "Your encryption passphrase" | cryptsetup -q luksFormat --type luks2 --cipher aes-xts-plain64 --hash sha256 --key-size 256 --pbkdf argon2i /dev/loop7 -
Open the encrypted device
- $ echo -n "Your encryption passphrase" | cryptsetup luksOpen /dev/loop7 backing-encrypted -
Dump the original data to the encrypted device
- $ dd if=backing of=/dev/mapper/backing-encrypted
Create another BackingImage with the new image file backing-encrypted
- Name: parrot-encrypted
- Type: upload

Create StorageClass with the same secret key

apiVersion: v1
kind: Secret
metadata:
  name: longhorn-crypto
  namespace: longhorn-system
stringData:
  CRYPTO_KEY_VALUE: "Your encryption passphrase"
  CRYPTO_KEY_PROVIDER: "secret"
  CRYPTO_KEY_CIPHER: "aes-xts-plain64"
  CRYPTO_KEY_HASH: "sha256"
  CRYPTO_KEY_SIZE: "256"
  CRYPTO_PBKDF: "argon2i"

---

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: longhorn-parrot-encrypted
provisioner: driver.longhorn.io
allowVolumeExpansion: true
parameters:
  numberOfReplicas: "2"
  staleReplicaTimeout: "2880" # 48 hours in minutes
  fromBackup: ""
  encrypted: "true"
  backingImage: "parrot-encrypted"
  backingImageDataSourceType: "upload"
  csi.storage.k8s.io/provisioner-secret-name: "longhorn-crypto"
  csi.storage.k8s.io/provisioner-secret-namespace: "longhorn-system"
  csi.storage.k8s.io/node-publish-secret-name: "longhorn-crypto"
  csi.storage.k8s.io/node-publish-secret-namespace: "longhorn-system"
  csi.storage.k8s.io/node-stage-secret-name: "longhorn-crypto"
  csi.storage.k8s.io/node-stage-secret-namespace: "longhorn-system"

Create the PVC

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: longhorn-volume-parrot-encrypted
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn-parrot-encrypted
  resources:
    requests:
      storage: 5Gi

Create the Pod with the encrypted Volume

apiVersion: v1
kind: Pod
metadata:
  name: longhorn-simple-pod
  namespace: default
spec:
  nodeSelector:
    kubernetes.io/hostname: kworker1
  restartPolicy: Always
  containers:
    - name: ubuntu
      image: ubuntu:22.04
      imagePullPolicy: IfNotPresent
      command: [ "/bin/bash", "-c", "--" ]
      args: [ "while true; do sleep 30; done;" ]
      volumeMounts:
        - name: vol
          mountPath: /data
      ports:
        - containerPort: 80
  volumes:
    - name: vol
      persistentVolumeClaim:
        claimName: longhorn-volume-parrot-encrypted

Results

The pod can mount the volume
The data content is the same as the original
The device is encrypted

cc @derekbit @WebberHuang1118 @shuo-wu

ChanYiLin · 2024-06-14T09:08:06Z

TODO

LEP: feat(lep): backing image clone and encryption support #8740
backing-image-manager PR: feat(backing encryption): backing image encryption support backing-image-manager#251
longhorn-manager PR: feat(backingimage): backing image clone and encryption support longhorn-manager#2876
longhorn CRD PR: feat(backingimage): add clone source type and secret to the backing image #8790
~~- [ ] generate longhorn manager API client: TBD~~

longhorn-io-github-bot · 2024-06-19T01:04:27Z

Pre Ready-For-Testing Checklist

Where is the reproduce steps/test steps documented?
The reproduce steps/test steps are at:

Raw Image Encryption

Create BackingImage

apiVersion: longhorn.io/v1beta2
kind: BackingImage
metadata:
    name: parrot
    namespace: longhorn-system
spec:
    sourceType: download
    sourceParameters:
        url: https://longhorn-backing-image.s3-us-west-1.amazonaws.com/parrot.raw
    checksum: 304f3ed30ca6878e9056ee6f1b02b328239f0d0c2c1272840998212f9734b196371560b3b939037e4f4c2884ce457c2cbc9f0621f4f5d1ca983983c8cdf8cd9a

Create Secret and StorageClass

apiVersion: v1
kind: Secret
metadata:
  name: longhorn-crypto
  namespace: longhorn-system
stringData:
  CRYPTO_KEY_VALUE: "Your encryption passphrase"
  CRYPTO_KEY_PROVIDER: "secret"
  CRYPTO_KEY_CIPHER: "aes-xts-plain64"
  CRYPTO_KEY_HASH: "sha256"
  CRYPTO_KEY_SIZE: "256"
  CRYPTO_PBKDF: "argon2i"
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: longhorn-crypto-global
provisioner: driver.longhorn.io
allowVolumeExpansion: true
parameters:
  numberOfReplicas: "2"
  staleReplicaTimeout: "2880" # 48 hours in minutes
  fromBackup: ""
  encrypted: "true"
  backingImage: "parrot-cloned-encrypted"
  backingImageDataSourceType: "clone"
  csi.storage.k8s.io/provisioner-secret-name: "longhorn-crypto"
  csi.storage.k8s.io/provisioner-secret-namespace: "longhorn-system"
  csi.storage.k8s.io/node-publish-secret-name: "longhorn-crypto"
  csi.storage.k8s.io/node-publish-secret-namespace: "longhorn-system"
  csi.storage.k8s.io/node-stage-secret-name: "longhorn-crypto"
  csi.storage.k8s.io/node-stage-secret-namespace: "longhorn-system"

Create Encrypted Backing Image

apiVersion: longhorn.io/v1beta2
kind: BackingImage
metadata:
  name: parrot-cloned-encrypted
  namespace: longhorn-system
spec:
  sourceType: clone
  sourceParameters:
    backing-image: parrot
    encryption: encrypt
    secret: longhorn-crypto
    secret-namespace: longhorn-system

Create PVC

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: longhorn-backing-image-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn-crypto-global
  resources:
    requests:
      storage: 1Gi

Create Pod to use the encrypted Volume with the encrypted BackingImage

apiVersion: v1
kind: Pod
metadata:
  name: longhorn-simple-pod
  namespace: default
spec:
  restartPolicy: Always
  containers:
    - name: ubuntu
      image: ubuntu:22.04
      imagePullPolicy: IfNotPresent
      command: [ "/bin/bash", "-c", "--" ]
      args: [ "while true; do sleep 30; done;" ]
      volumeMounts:
        - name: vol
          mountPath: /data
      ports:
        - containerPort: 80
  volumes:
    - name: vol
      persistentVolumeClaim:
        claimName: longhorn-backing-image-pvc

Decrypt the Encrypted BackingImage

apiVersion: longhorn.io/v1beta2
kind: BackingImage
metadata:
  name: parrot-cloned-decrypt
  namespace: longhorn-system
spec:
  sourceType: clone
  sourceParameters:
    backing-image: parrot-cloned-encrypted
    encryption: decrypt
    secret: longhorn-crypto
    secret-namespace: longhorn-system

The checksum of the parrot-cloned-decrypt should be the same as parrot

Qcow2 Image Encryption

Follow the same process, but change the BackingImage to qcow2
- https://longhorn-backing-image.s3-us-west-1.amazonaws.com/parrot.qcow2

PRs

Does the PR include deployment change (YAML/Chart)? If so, where are the PRs for both YAML file and Chart?
- longhorn CRD PR: feat(backingimage): add clone source type and secret to the backing image #8790
Have the backend code been merged (Manager, Engine, Instance Manager, BackupStore etc) (including backport-needed/*)?
The PR is at
- backing-image-manager PR: feat(backing encryption): backing image encryption support backing-image-manager#251
- longhorn-manager PR: feat(backingimage): backing image clone and encryption support longhorn-manager#2876
If labeled: require/LEP Has the Longhorn Enhancement Proposal PR submitted?
The LEP PR is at
- LEP: feat(lep): backing image clone and encryption support #8740
If labeled: area/ui Has the UI issue filed or ready to be merged (including backport-needed/*)?
The UI issue/PR is at
- [UI][FEATURE] Add backing image encryption and clone support #8789
If labeled: require/doc Has the necessary document PR submitted or merged (including backport-needed/*)?
The documentation issue/PR is at
- Lh7051 backingimage encryption support website#927

roger-ryao · 2024-07-08T08:26:46Z

Verified on master-head 20240708

longhorn master-head e5a1b58
backing-image-manager master-head longhorn/backing-image-manager@dfbe0d0
longhorn-manager master-head longhorn/longhorn-manager@754a93e
longhorn-ui master-head longhorn/longhorn-ui@546becd

The test steps
#7051 (comment)

Result passed

Testing Items via CRD and UI

Raw Image Encryption
1. Create an encrypted backing Image parrot-cloned-encrypted cloned from parrot.raw
2. Decrypt the Encrypted BackingImage parrot-cloned-decrypt from parrot-cloned-encrypted
3. Clone Backing Image from parrot.raw
QCOW2 Image Encryption
1. Create an encrypted backing Image parrot-cloned-encrypted clone from parrot.qcow2
2. Decrypt the Encrypted BackingImage parrot-cloned-decrypt from parrot-cloned-encrypted
3. Clone Backing Image from parrot.qcow2
require/lep
- feat(lep): backing image clone and encryption support #8740
require/ui
- [UI][FEATURE] Add backing image encryption and clone support #8789
require/auto-e2e-test
- [TEST][FEATURE] Support volume encryption for (encrypted) backing image volumes #7052
require/doc
- Lh7051 backingimage encryption support website#927

innobead added kind/feature Feature request, new feature require/auto-e2e-test Require adding/updating auto e2e test cases if they can be automated require/doc Require updating the longhorn.io documentation require/lep Require adding/updating enhancement proposal labels Nov 6, 2023

innobead added this to the v1.7.0 milestone Nov 6, 2023

innobead added priority/0 Must be implement or fixed in this release (managed by PO) area/security System or volume data access security area/backing-image Backing image related area/volume-encryption Volume encryption related labels Nov 6, 2023

github-actions bot mentioned this issue Nov 6, 2023

[TEST][FEATURE] Support volume encryption for (encrypted) backing image volumes #7052

Open

innobead added the area/virtualization label Nov 6, 2023

innobead assigned ChanYiLin Apr 10, 2024

ChanYiLin added the highlight Important feature/issue to highlight label May 6, 2024

longhorn deleted a comment from longhorn-io-github-bot May 8, 2024

derekbit added the require/ui Require adding functionality to UI label Jun 20, 2024

github-actions bot mentioned this issue Jun 20, 2024

[UI][FEATURE] Support volume encryption for (encrypted) backing image volumes #8788

Closed

derekbit assigned roger-ryao Jul 6, 2024

roger-ryao closed this as completed Jul 8, 2024

derekbit added this to Longhorn Sprint Aug 3, 2024

derekbit moved this to Closed in Longhorn Sprint Aug 3, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FEATURE] Support volume encryption for (encrypted) backing image volumes #7051

[FEATURE] Support volume encryption for (encrypted) backing image volumes #7051

innobead commented Nov 6, 2023 •

edited

Loading

ChanYiLin commented May 8, 2024 •

edited

Loading

ChanYiLin commented May 8, 2024

ChanYiLin commented May 8, 2024

ChanYiLin commented May 10, 2024 •

edited

Loading

ChanYiLin commented Jun 14, 2024 •

edited

Loading

longhorn-io-github-bot commented Jun 19, 2024 •

edited by roger-ryao

Loading

roger-ryao commented Jul 8, 2024

[FEATURE] Support volume encryption for (encrypted) backing image volumes #7051

[FEATURE] Support volume encryption for (encrypted) backing image volumes #7051

Comments

innobead commented Nov 6, 2023 • edited Loading

Is your feature request related to a problem? Please describe (👍 if you like this request)

Describe the solution you'd like

Describe alternatives you've considered

Additional context

ChanYiLin commented May 8, 2024 • edited Loading

Experiments

Create Encrypted Volume with BackingImage - FAILED

ChanYiLin commented May 8, 2024

Experiments

Create Encrypted Volume with Encrypted BackingImage - SUCCEEDED

ChanYiLin commented May 8, 2024

Other Reference

AWS

Ceph

ChanYiLin commented May 10, 2024 • edited Loading

Experiments

Manually Encrypt a Non-encrypted Raw Backing Image - SUCCEEDED

Results

ChanYiLin commented Jun 14, 2024 • edited Loading

TODO

longhorn-io-github-bot commented Jun 19, 2024 • edited by roger-ryao Loading

Pre Ready-For-Testing Checklist

Raw Image Encryption

Qcow2 Image Encryption

PRs

roger-ryao commented Jul 8, 2024

innobead commented Nov 6, 2023 •

edited

Loading

ChanYiLin commented May 8, 2024 •

edited

Loading

ChanYiLin commented May 10, 2024 •

edited

Loading

ChanYiLin commented Jun 14, 2024 •

edited

Loading

longhorn-io-github-bot commented Jun 19, 2024 •

edited by roger-ryao

Loading