Skip to content

Commit

Permalink
Merge pull request hwameistor#218 from sun7927/minio
Browse files Browse the repository at this point in the history 
updated the readme of the minio solution
  • Loading branch information
@sun7927
sun7927 authored Aug 22, 2022
2 parents 6908192 + b905208 commit 7fc0186
Showing 1 changed file with 64 additions and 62 deletions.
126 changes: 64 additions & 62 deletions .../minio/HwameiStor-MinIOSolutionReadMe.txt → deploy/solutions/minio/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,34 +1,35 @@

HwameiStor-MinIO Cloud Native Solution
# HwameiStor-MinIO Cloud Native Solution

-- Solution Introduction
## Solution Introduction

HwameiStor is a high availability local storage system for cloud-native stateful workloads. It creates a local storage resource pool for centrally managing all disks such as HDD, SSD, and NVMe. It uses the CSI architecture to provide distributed services with local volumes and provides data persistence capabilities for stateful cloud-native workloads or components.

MinIO is a High Performance Object Storage released under GNU Affero General Public License v3.0. It is API-compatible with Amazon S3 cloud storage service， and can be used to build high performance infrastructure for machine learning, analytics and application data workloads.


MinIO is usually used as the main storage for cloud-native applications which require higher throughput and lower latency. MinIO's read/write performance can achieve up to the speed of 183 GB/sec and 171 GB/sec.

The ultimate high performance of MinIO is inseparable from the underlying storage platform. Local storage has been recognized as the highest r/w performance protocol among many other storage protocols, which can undoubtedly provide performance guarantee for MinIO. HwameiStor is such a local storage system that waves with the cloud-native era. High performance, high availability, automation, low cost, rapid deployment, and more benefits attract you to replace those expensive traditional storage schemes such as SAN (Storage Area Network).


-- Solution Validation
## Solution Validation

Testing Environment

This test uses three virtual machine nodes to deploy a Kubernetes cluster: 1 Master + 3 Worker nodes, and the kubelet version is 1.22.0.

[root@k8s-10-6-163-52 minio-test1]# kubectl get no
NAME STATUS ROLES AGE VERSION
k8s-10-6-163-51 Ready <none> 80d v1.21.0
k8s-10-6-163-52 Ready control-plane,master 80d v1.21.0
k8s-10-6-163-53 Ready <none> 80d v1.21.0
k8s-10-6-163-54 Ready <none> 23d v1.21.0
```console
$ kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-master-1 Ready master 96d v1.24.3-2+63243a96d1c393
k8s-worker-1 Ready worker 96h v1.24.3-2+63243a96d1c393
k8s-worker-2 Ready worker 96h v1.24.3-2+63243a96d1c393
k8s-worker-3 Ready worker 96d v1.24.3-2+63243a96d1c393
```

Deploy HwameiStor local storage on Kubernetes.

[root@k8s-10-6-163-52 minio-test1]# kubectl get all -nhwameistor
```console
# kubectl get all -nhwameistor
NAME READY STATUS RESTARTS AGE
pod/hwameistor-admission-controller-56bbc5c9fc-5bptb 1/1 Running 1 45h
pod/hwameistor-local-disk-csi-controller-c7bdffcff-tnmmh 2/2 Running 272 45h
Expand All @@ -43,6 +44,7 @@ pod/hwameistor-local-storage-gwx9b 2/2 Running
pod/hwameistor-local-storage-p2q7r 2/2 Running 28 45h
pod/hwameistor-scheduler-68dc49bd69-hh4b8 1/1 Running 124 45h


NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/hwameistor-admission-controller ClusterIP 10.108.62.244 <none> 443/TCP 45h
service/local-disk-manager-metrics ClusterIP 10.109.190.29 <none> 8383/TCP,8686/TCP 45h
Expand All @@ -62,55 +64,58 @@ replicaset.apps/hwameistor-admission-controller-56bbc5c9fc 1 1
replicaset.apps/hwameistor-local-disk-csi-controller-c7bdffcff 1 1 1 45h
replicaset.apps/hwameistor-local-storage-csi-controller-86d55d6bdc 1 1 1 45h
replicaset.apps/hwameistor-scheduler-68dc49bd69 1 1 1 45h
```

View local storage disks status

[root@k8s-10-6-163-52 minio-test1]# kubectl get ld
NAME NODEMATCH CLAIM PHASE
k8s-10-6-163-51-sda k8s-10-6-163-51 Inuse
k8s-10-6-163-51-sdb k8s-10-6-163-51 Claimed
k8s-10-6-163-51-sdc k8s-10-6-163-51 Claimed
k8s-10-6-163-51-sdd k8s-10-6-163-51 Claimed
k8s-10-6-163-51-sde k8s-10-6-163-51 Claimed
k8s-10-6-163-51-sdf k8s-10-6-163-51 Claimed
k8s-10-6-163-52-sda k8s-10-6-163-52 Inuse
k8s-10-6-163-52-sdb k8s-10-6-163-52 Unclaimed
k8s-10-6-163-53-sda k8s-10-6-163-53 Inuse
k8s-10-6-163-53-sdb k8s-10-6-163-53 Claimed
k8s-10-6-163-53-sdc k8s-10-6-163-53 Claimed
k8s-10-6-163-53-sdd k8s-10-6-163-53 Claimed
k8s-10-6-163-53-sde k8s-10-6-163-53 Claimed
k8s-10-6-163-53-sdf k8s-10-6-163-53 Claimed
k8s-10-6-163-54-sda k8s-10-6-163-54 Inuse
k8s-10-6-163-54-sdb k8s-10-6-163-54 Unclaimed
k8s-10-6-163-54-sdc k8s-10-6-163-54 Unclaimed
k8s-10-6-163-54-sdd k8s-10-6-163-54 Unclaimed
k8s-10-6-163-54-sde k8s-10-6-163-54 Unclaimed
k8s-10-6-163-54-sdf k8s-10-6-163-54 Unclaimed

```console
# kubectl get ld
NAME NODEMATCH CLAIM PHASE
k8s-master-1-sda k8s-master-1 Inuse
k8s-master-1-sdb k8s-master-1 Claimed
k8s-master-1-sdc k8s-master-1 Claimed
k8s-master-1-sdd k8s-master-1 Claimed
k8s-master-1-sde k8s-master-1 Claimed
k8s-master-1-sdf k8s-master-1 Claimed
k8s-worker-1-sda k8s-worker-1 Inuse
k8s-worker-1-sdb k8s-worker-1 Unclaimed
k8s-worker-2-sda k8s-worker-2 Inuse
k8s-worker-2-sdb k8s-worker-2 Claimed
k8s-worker-2-sdc k8s-worker-2 Claimed
k8s-worker-2-sdd k8s-worker-2 Claimed
k8s-worker-2-sde k8s-worker-2 Claimed
k8s-worker-2-sdf k8s-worker-2 Claimed
k8s-worker-3-sda k8s-worker-3 Inuse
k8s-worker-3-sdb k8s-worker-3 Unclaimed
k8s-worker-3-sdc k8s-worker-3 Unclaimed
k8s-worker-3-sdd k8s-worker-3 Unclaimed
k8s-worker-3-sde k8s-worker-3 Unclaimed
k8s-worker-3-sdf k8s-worker-3 Unclaimed
```

View StorageClass status

[root@k8s-10-6-163-52 minio-test1]# kubectl get sc
```console
# kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
hwameistor-storage-lvm-hdd lvm.hwameistor.io Delete WaitForFirstConsumer true 45h
```



Deploy MinIO on Kubernetes

## Deploy MinIO on Kubernetes

Deploy based on official helm chart and install MinIO chart.

[root@k8s-10-6-163-52 minio-test1]# helm repo add minio https://helm.min.io
```console
$ helm repo add minio <https://helm.min.io>

[root@k8s-10-6-163-52 minio-test1]# helm repo list | grep minio
minio https://helm.min.io/
$ helm repo list | grep minio
minio <https://helm.min.io/>
```

### Standalone mode deployment

Standalone mode deployment.

helm install minio-2 \
```console
$ helm install minio-2 \
--namespace minio-2 --create-namespace \
--set accessKey=admin,secretKey=admin123 \
--set mode=standalone \
Expand All @@ -120,7 +125,7 @@ helm install minio-2 \
--set persistence.storageClass=hwameistor-storage-lvm-hdd \
minio/minio

[root@k8s-10-6-163-52 minio-test1]# kubectl get all -nminio-2
$ kubectl get all -nminio-2
NAME READY STATUS RESTARTS AGE
pod/minio-2-785f5c9985-7f5pf 1/1 Running 0 97m

Expand All @@ -132,18 +137,20 @@ deployment.apps/minio-2 1/1 1 1 97m

NAME DESIRED CURRENT READY AGE
replicaset.apps/minio-2-785f5c9985 1 1 1 97m

```

View PVCs on HwameiStor

[root@k8s-10-6-163-52 minio-test1]# kubectl get pvc -nminio-2
```console
$ kubectl get pvc -nminio-2
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
minio-2 Bound pvc-3d4c1846-fc64-4af0-8104-64fc66c1c1bb 2Gi RWO hwameistor-storage-lvm-hdd 103m
```

### Distributed mode deployment

Distributed mode deployment

helm install minio-1 \
```console
$ helm install minio-1 \
--namespace minio --create-namespace \
--set accessKey=admin,secretKey=admin123 \
--set mode=distributed \
Expand All @@ -153,7 +160,7 @@ helm install minio-1 \
--set persistence.storageClass=hwameistor-storage-lvm-hdd \
minio/minio

[root@k8s-10-6-163-52 minio-test1]# kubectl get all -nminio
$ kubectl get all -nminio
NAME READY STATUS RESTARTS AGE
pod/minio-1-0 1/1 Running 0 13h
pod/minio-1-1 1/1 Running 0 13h
Expand All @@ -166,20 +173,15 @@ service/minio-1-svc ClusterIP None <none> 9000/TCP

NAME READY AGE
statefulset.apps/minio-1 4/4 13h

```

View PVCs on HwameiStor

[root@k8s-10-6-163-52 minio-test1]# kubectl get pvc -nminio
```console
$ kubectl get pvc -nminio
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
export-minio-1-0 Bound pvc-bfbff95b-1afc-4484-8039-6ae402fd9116 2Gi RWO hwameistor-storage-lvm-hdd 13h
export-minio-1-1 Bound pvc-fc178030-1dde-4d14-90db-796078041ae2 2Gi RWO hwameistor-storage-lvm-hdd 13h
export-minio-1-2 Bound pvc-527cc3af-7fa4-4496-b4fc-08d69166d582 2Gi RWO hwameistor-storage-lvm-hdd 13h
export-minio-1-3 Bound pvc-29ff7a1c-5097-4e84-ac04-961eb735ddec 2Gi RWO hwameistor-storage-lvm-hdd 13h







```

0 comments on commit 7fc0186

Please sign in to comment.