Longhorn

Longhorn is a distributed block storage system for Kubernetes. Longhorn is lightweight, reliable, and easy-to-use. You can deploy Longhorn on an existing Kubernetes cluster with one simple command. Once Longhorn is deployed, it adds persistent volume support to the Kubernetes cluster.

Longhorn implements distributed block storage using containers and microservices. Longhorn creates a dedicated storage controller for each block device volume and sychronously replicates the volume across multiple replicas stored on multiple nodes. The storage controller and replicas are themselves orchestrated using Kubernetes. Longhorn supports snapshots, backups, and even allows you to schedule recurring snapshots and backups!

You can read more details of Longhorn and its design here.

Longhorn is a work in progress. We appreciate your comments as we continue to work on it!

Source Code

Longhorn is 100% open source software. Project source code is spread across a number of repos:

Longhorn Engine -- Core controller/replica logic https://github.com/rancher/longhorn-engine
Longhorn Manager -- Longhorn orchestration, includes Flexvolume driver for Kubernetes https://github.com/rancher/longhorn-manager
Longhorn UI -- Dashboard https://github.com/rancher/longhorn-ui

Demo

Requirements

Minimal Requirements

Docker v1.13+
Kubernetes v1.8+
Make sure open-iscsi has been installed in all nodes of the Kubernetes cluster. For GKE, recommended Ubuntu as guest OS image since it contains open-iscsi already.

Kubernetes Driver Requirements

Longhorn can be used in Kubernetes to provide persistent storage through either Longhorn Container Storage Interface (CSI) driver or Longhorn Flexvolume driver. Longhorn will automatically deploy one of the drivers, depends on user's Kubernetes cluster's setup. User can also specify the driver in the deployment yaml file. CSI is preferred.

Requirement for the CSI driver

Kubernetes v1.10+
1. CSI is in beta release for this version of Kubernetes, and enabled by default.
Mount Propagation feature gate enabled.
1. It's enabled by default in Kubernetes v1.10. But some early versions of RKE may not enable it.
If above conditions cannot be met, Longhorn will falls back to use Flexvolume driver.

Check if your setup satisfied CSI requirement

Use the following command to check your Kubernetes server version

# kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.3", GitCommit:"2bba0127d85d5a46ab4b778548be28623b32d0b0", GitTreeState:"clean", BuildDate:"2018-05-21T09:17:39Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.1", GitCommit:"d4ab47518836c750f9949b9e0d387f20fb92260b", GitTreeState:"clean", BuildDate:"2018-04-12T14:14:26Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

The Server Version should be v1.10 or above.

Use the following command on the hosts to check if the feature gate is enabled for Mount Propagation

# ps aux|grep kube|grep MountPropagation
root      1707  3.1 12.4 1087008 503848 ?      Ssl  Jul12 1288:35 kube-apiserver --storage-backend=etcd3 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --apiserver-count=1 --secure-port=6443 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --allow-privileged=true --insecure-port=0 --admission-control=ServiceAccount,NamespaceLifecycle,LimitRanger,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota,DefaultTolerationSeconds --cloud-provider= --service-cluster-ip-range=10.43.0.0/16 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-account-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --feature-gates=MountPropagation=true --insecure-bind-address=127.0.0.1 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-servers=https://138.197.199.191:2379 --etcd-prefix=/registry
root      1760  4.7  6.4 1508564 260724 ?      Ssl  Jul12 1970:59 kubelet --network-plugin=cni --resolv-conf=/etc/resolv.conf --cluster-domain=cluster.local --v=2 --enforce-node-allocatable= --cgroups-per-qos=True --cni-bin-dir=/opt/cni/bin --cluster-dns=10.43.0.10 --cloud-provider= --fail-swap-on=false --address=0.0.0.0 --cadvisor-port=0 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --hostname-override=yasker-longhorn-dev-1 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --root-dir=/var/lib/kubelet --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --feature-gates=MountPropagation=true --cni-conf-dir=/etc/cni/net.d --allow-privileged=true --pod-infra-container-image=rancher/pause-amd64:3.0 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --read-only-port=0 --anonymous-auth=false --cgroup-driver=cgroupfs

Both kube-apiserver and kubelet should have --feature-gates=MountPropagation=true

Requirement for the Flexvolume driver

Kubernetes v1.8+
Make sure curl, findmnt, grep, awk and blkid has been installed in the every node of the Kubernetes cluster.
User need to know the volume plugin directory in order to setup the driver correctly.
1. Rancher RKE: /var/lib/kubelet/volumeplugins
2. Google GKE: /home/kubernetes/flexvolume
3. For other distro, please find the correct directory by running ps aux|grep kubelet on the host and check the --volume-plugin-dir parameter. If there is none, it would be the default value /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ .

Upgrading

For instructions on how to upgrade Longhorn v0.1 or v0.2 to v0.3, see this document.

Deployment

Create the deployment of Longhorn in your Kubernetes cluster is easy.

If you're using Rancher RKE, or other distro with Kubernetes v1.10+ and Mount Propagation enabled, you can just do:

kubectl apply -f https://raw.githubusercontent.com/rancher/longhorn/v0.3-rc/deploy/longhorn.yaml

If you're using Flexvolume driver with other Kubernetes Distro, replace the value of $FLEXVOLUME_DIR in the following command with your own Flexvolume Directory as specified above.

FLEXVOLUME_DIR="/home/kubernetes/flexvolume/"
curl -s https://raw.githubusercontent.com/rancher/longhorn/v0.3-rc/deploy/longhorn.yaml|sed "s#^\( *\)value: \"/var/lib/kubelet/volumeplugins\"#\1value: \"${FLEXVOLUME_DIR}\"#g" > longhorn.yaml
kubectl apply -f longhorn.yaml

For Google Kubernetes Engine (GKE) users, see here before proceed.

Longhorn Manager and Longhorn Driver will be deployed as daemonsets in a separate namespace called longhorn-system, as you can see in the yaml file.

When you see those pods has started correctly as follows, you've deployed the Longhorn successfully.

Deployed with CSI driver:

# kubectl -n longhorn-system get pod
NAME                                        READY     STATUS    RESTARTS   AGE
csi-attacher-0                              1/1       Running   0          6h
csi-provisioner-0                           1/1       Running   0          6h
engine-image-ei-57b85e25-8v65d              1/1       Running   0          7d
engine-image-ei-57b85e25-gjjs6              1/1       Running   0          7d
engine-image-ei-57b85e25-t2787              1/1       Running   0          7d
longhorn-csi-plugin-4cpk2                   2/2       Running   0          6h
longhorn-csi-plugin-ll6mq                   2/2       Running   0          6h
longhorn-csi-plugin-smlsh                   2/2       Running   0          6h
longhorn-driver-deployer-7b5bdcccc8-fbncl   1/1       Running   0          6h
longhorn-manager-7x8x8                      1/1       Running   0          6h
longhorn-manager-8kqf4                      1/1       Running   0          6h
longhorn-manager-kln4h                      1/1       Running   0          6h
longhorn-ui-f849dcd85-cgkgg                 1/1       Running   0          5d

Or with Flexvolume driver

# kubectl -n longhorn-system get pod
NAME                                        READY     STATUS    RESTARTS   AGE
engine-image-ei-57b85e25-8v65d              1/1       Running   0          7d
engine-image-ei-57b85e25-gjjs6              1/1       Running   0          7d
engine-image-ei-57b85e25-t2787              1/1       Running   0          7d
longhorn-driver-deployer-5469b87b9c-b9gm7   1/1       Running   0          2h
longhorn-flexvolume-driver-lth5g            1/1       Running   0          2h
longhorn-flexvolume-driver-tpqf7            1/1       Running   0          2h
longhorn-flexvolume-driver-v9mrj            1/1       Running   0          2h
longhorn-manager-7x8x8                      1/1       Running   0          9h
longhorn-manager-8kqf4                      1/1       Running   0          9h
longhorn-manager-kln4h                      1/1       Running   0          9h
longhorn-ui-f849dcd85-cgkgg                 1/1       Running   0          5d

Access the UI

Use kubectl -n longhorn-system get svc to get the external service IP for UI:

NAME                TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)        AGE
longhorn-backend    ClusterIP      10.20.248.250   <none>           9500/TCP       58m
longhorn-frontend   LoadBalancer   10.20.245.110   100.200.200.123   80:30697/TCP   58m

If the Kubernetes Cluster supports creating LoadBalancer, user can then use EXTERNAL-IP(100.200.200.123 in the case above) of longhorn-frontend to access the Longhorn UI. Otherwise the user can use <node_ip>:<port> (port is 30697in the case above) to access the UI.

Longhorn UI would connect to the Longhorn Manager API, provides the overview of the system, the volume operations, and the snapshot/backup operations. It's highly recommended for the user to check out Longhorn UI.

Notice the current UI is unauthenticated.

Use the Longhorn with Kubernetes

Longhorn provides persistent volume directly to Kubernetes through one of the Longhorn drivers. No matter which driver you're using, you can use Kubernetes StorageClass to provision your persistent volumes.

Use following command to create a default Longhorn StorageClass named longhorn.

kubectl apply -f https://raw.githubusercontent.com/rancher/longhorn/v0.3-rc/examples/storageclass.yaml

Now you can create a pod using Longhorn like this:

kubectl apply -f https://raw.githubusercontent.com/rancher/longhorn/v0.3-rc/examples/pvc.yaml

The yaml contains two parts:

Create a PVC using Longhorn StorageClass.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: longhorn-volv-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 2Gi

Use it in the a Pod as a persistent volume:

apiVersion: v1
kind: Pod
metadata:
  name: volume-test
  namespace: default
spec:
  containers:
  - name: volume-test
    image: nginx:stable-alpine
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - name: volv
      mountPath: /data
    ports:
    - containerPort: 80
  volumes:
  - name: volv
    persistentVolumeClaim:
      claimName: longhorn-volv-pvc

More examples are available at ./examples/

Feature Usage

Snapshot

A snapshot in Longhorn represents a volume state at a given time, stored in the same location of volume data on physical disk of the host. Snapshot creation is instant in Longhorn.

User can revert to any previous taken snapshot using the UI. Since Longhorn is a distributed block storage, please make sure the Longhorn volume is umounted from the host when revert to any previous snapshot, otherwise it will confuse the node filesystem and cause corruption.

Backup

A backup in Longhorn represents a volume state at a given time, stored in the BackupStore which is outside of the Longhorn System. Backup creation will involving copying the data through the network, so it will take time.

A corresponding snapshot is needed for creating a backup. And user can choose to backup any snapshot previous created.

A BackupStore is a NFS server or S3 compatible server.

A BackupTarget represents a BackupStore in the Longhorn System. The BackupTarget can be set at Settings/General/BackupTarget

If user is using a S3 compatible server as the BackupTarget, the BackupTargetSecret is needed for authentication informations. User need to manually create it as a Kubernetes Secret in the longhorn-system namespace. See below for details.

Setup a testing backupstore

We provides two testing purpose backupstore based on NFS server and Minio S3 server for testing, in ./deploy/backupstores.

Use following command to setup a Minio S3 server for BackupStore after longhorn-system was created.

kubectl apply -f https://raw.githubusercontent.com/rancher/longhorn/v0.3-rc/deploy/backupstores/minio-backupstore.yaml

Now set Settings/General/BackupTarget to

s3://backupbucket@us-east-1/backupstore

And Setttings/General/BackupTargetSecret to

minio-secret

Click the Backup tab in the UI, it should report an empty list without error out.

The minio-secret yaml looks like this:

apiVersion: v1
kind: Secret
metadata:
  name: minio-secret
  namespace: longhorn-system
type: Opaque
data:
  AWS_ACCESS_KEY_ID: bG9uZ2hvcm4tdGVzdC1hY2Nlc3Mta2V5 # longhorn-test-access-key
  AWS_SECRET_ACCESS_KEY: bG9uZ2hvcm4tdGVzdC1zZWNyZXQta2V5 # longhorn-test-secret-key
  AWS_ENDPOINTS: aHR0cDovL21pbmlvLXNlcnZpY2UuZGVmYXVsdDo5MDAw # http://minio-service.default:9000

Notice the secret must be created in the longhorn-system namespace for Longhorn to access.

Recurring Snapshot and Backup

Longhorn supports recurring snapshot and backup for volumes. User only need to set when he/she wish to take the snapshot and/or backup, and how many snapshots/backups needs to be retains, then Longhorn will automatically create snapshot/backup for the user at that time, as long as the volume is attached to a node.

User can find the setting for the recurring snapshot and backup in the Volume Detail page.

Uninstall Longhorn

Longhorn CRD has finalizers in them, so user should delete the volumes and related resource first, give manager a chance to clean up after them.

1. Clean up volume and related resources

kubectl -n longhorn-system delete volumes.longhorn.rancher.io --all

Check the result using:

kubectl -n longhorn-system get volumes.longhorn.rancher.io
kubectl -n longhorn-system get engines.longhorn.rancher.io
kubectl -n longhorn-system get replicas.longhorn.rancher.io

Make sure all reports No resources found. before continuing.

2. Clean up engine images and nodes

kubectl -n longhorn-system delete engineimages.longhorn.rancher.io --all
kubectl -n longhorn-system delete nodes.longhorn.rancher.io --all

Check the result using:

kubectl -n longhorn-system get engineimages.longhorn.rancher.io
kubectl -n longhorn-system get nodes.longhorn.rancher.io

Make sure all reports No resources found. before continuing.

3. Uninstall Longhorn System

kubectl delete -f https://raw.githubusercontent.com/rancher/longhorn/v0.3-rc/deploy/longhorn.yaml

License

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Name		Name	Last commit message	Last commit date
Latest commit Cannot retrieve latest commit at this time. History 88 Commits
deploy		deploy
docs		docs
examples		examples
scripts		scripts
LICENSE		LICENSE
README.md		README.md

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

Longhorn

Source Code

Demo

Requirements

Minimal Requirements

Kubernetes Driver Requirements

Requirement for the CSI driver

Check if your setup satisfied CSI requirement

Requirement for the Flexvolume driver

Upgrading

Deployment

Access the UI

Use the Longhorn with Kubernetes

Feature Usage

Snapshot

Backup

Setup a testing backupstore

Recurring Snapshot and Backup

Other topics

Upgrade from v0.2

Multiple disks support

iSCSI support

Google Kubernetes Engine

Troubleshotting

Restoring Stateful Set volumes

Base Image support

Uninstall Longhorn

1. Clean up volume and related resources

2. Clean up engine images and nodes

3. Uninstall Longhorn System

License

About

Releases

Packages

Languages

License

Haribo112/longhorn

Folders and files

Latest commit

History

Repository files navigation

Longhorn

Source Code

Demo

Requirements

Minimal Requirements

Kubernetes Driver Requirements

Requirement for the CSI driver

Check if your setup satisfied CSI requirement

Requirement for the Flexvolume driver

Upgrading

Deployment

Access the UI

Use the Longhorn with Kubernetes

Feature Usage

Snapshot

Backup

Setup a testing backupstore

Recurring Snapshot and Backup

Other topics

Uninstall Longhorn

1. Clean up volume and related resources

2. Clean up engine images and nodes

3. Uninstall Longhorn System

License

About

Resources

License

Code of conduct

Stars

Watchers

Forks

Languages