Signed-off-by: mittachaitu sai.chaithanya@mayadata.io

What this PR does / why we need it:
This PR adds replica scale down support in the cstor-volume-mgmt(sidecar of cstor-istgt).

Note:

1. Only one replica at a time should be removed.

Pre-requisites:

1. All the cstorvolume replicas(CVR) should be in a Healthy state except the cstorvolume replica that is going to delete(i.e deleting CVR can be in any state).
2. There shouldn't be any ongoing scaleup process(Verify using cstorvolume CR replicationFactor should be equal to the desiredReplicationFactor).

Steps to execute by the user/operator:

1. The user will decrease the value of the desired replication factor and remove the replicaID entry from cstorvolume which will under cstorvolume.Spec.ReplicaDeatails by using Kubectl edit cstorvolume <cstorvolume_name> -n openebs(both should be updated at a time).
2. Upon successful removal of the replica user/operator will delete the corresponding CVR(CVR whose replicaID entry removed in step1).

User has the following PVC setup

sai@sai:~$ kubectl get pvc
NAME             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
demo-vol-claim   Bound    pvc-79ba260d-fad6-11e9-8791-42010a8000c5   5G         RWO            cstor-sc       10m

Following are cstorvolume and CVRs related to above PVC

sai@sai:~$ kubectl get cstorvolume -n openebs -l openebs.io/persistent-volume=pvc-79ba260d-fad6-11e9-8791-42010a8000c5
NAME                                       STATUS    AGE   CAPACITY
pvc-79ba260d-fad6-11e9-8791-42010a8000c5   Healthy   11m   5G

sai@sai:~$ kubectl get cvr -n openebs -l openebs.io/persistent-volume=pvc-79ba260d-fad6-11e9-8791-42010a8000c5
NAME                                                              USED   ALLOCATED   STATUS    AGE
pvc-79ba260d-fad6-11e9-8791-42010a8000c5-cstor-sparse-pool-7jsv   6K     6K          Healthy   11m
pvc-79ba260d-fad6-11e9-8791-42010a8000c5-cstor-sparse-pool-mch2   6K     6K          Healthy   11m
pvc-79ba260d-fad6-11e9-8791-42010a8000c5-cstor-sparse-pool-qa5i   6K     6K          Healthy   11m

Explanation for the above steps:

• Assume from above o/p if user need to delete pvc-79ba260d-fad6-11e9-8791-42010a8000c5-cstor-sparse-pool-mch2 CVR. User should follow the mentioned steps.
• Note the replicaID of above CVR under .spec.replicaID in this case replicaID is AEB135CD6AA1E65EF2DAEA085A4C9BA7.

Following is a snippet of cstorvolume:

apiVersion: openebs.io/v1alpha1
kind: CStorVolume
name: pvc-79ba260d-fad6-11e9-8791-42010a8000c5
spec:
    consistencyFactor: 2 ## Minimum no.of replicas required to say IO(read/write) is successfull
    desiredReplicationFactor: 3 ## Maximum no.of replicas are allowed to connect
    replicaDetails:
      ## knownReplicas contains map[string]string key as cvr replicaID and value as replica ZVOL_GUID
      knownReplicas:
        44253781C6BD1384DDF024C643B60E26: "6475964221803202402"
        AEB135CD6AA1E65EF2DAEA085A4C9BA7: "1440900316001731961"
        B2EEE0A0E4C9439926AD544D0E134D90: "8420817958580332064"
    replicationFactor: 3 ## Maximum no.of quorum replicas that can allowed to connect
    ...
    ...
  status:
    ...
    ...
    replicaDetails:
      knownReplicas:
        44253781C6BD1384DDF024C643B60E26: "6475964221803202402"
        AEB135CD6AA1E65EF2DAEA085A4C9BA7: "1440900316001731961"
        B2EEE0A0E4C9439926AD544D0E134D90: "8420817958580332064"

Above snippet will be updated as follows using kubectl edit

apiVersion: openebs.io/v1alpha1
kind: CStorVolume
name: pvc-79ba260d-fad6-11e9-8791-42010a8000c5
spec:
    consistencyFactor: 2
    desiredReplicationFactor: 2
    replicaDetails:
      knownReplicas:
        44253781C6BD1384DDF024C643B60E26: "6475964221803202402"
        B2EEE0A0E4C9439926AD544D0E134D90: "8420817958580332064"
    replicationFactor: 3
    ...
    ...
  status:
    ...
    ...
    replicaDetails:
      knownReplicas:
        44253781C6BD1384DDF024C643B60E26: "6475964221803202402"
        AEB135CD6AA1E65EF2DAEA085A4C9BA7: "1440900316001731961"
        B2EEE0A0E4C9439926AD544D0E134D90: "8420817958580332064"

After editing snippet 1 will looks as snippet 2( updated the desired replication factor to 2 and removed replicaID entry[AEB135CD...] from spec.replicaDetails.knownreplicas).

• Upon success removal events will be generated and spec & status of cstorvolume will be updated as follows(replica scaledown verification steps)
  Events on cstorvolume:

 Normal   Updated     12m (x2 over 12m)  pvc-79ba260d-fad6-11e9-8791-42010a8000c5-target-8c76ff8f6-ssw67, gke-sai-bdd-test-pool-1-775ad7d7-j5jj  Successfully updated the desiredReplicationFactor to 2

  spec:
    ....
    ....
    consistencyFactor: 2
    desiredReplicationFactor: 2
    replicaDetails:
      knownReplicas:
        44253781C6BD1384DDF024C643B60E26: "6475964221803202402"
        B2EEE0A0E4C9439926AD544D0E134D90: "8420817958580332064"
    replicationFactor: 2
  status:
    ...
    ...
    replicaDetails:
      knownReplicas:
        44253781C6BD1384DDF024C643B60E26: "6475964221803202402"
        B2EEE0A0E4C9439926AD544D0E134D90: "8420817958580332064"

If you observe status.replicadetails.knownreplicas is updated with latest info. ReplicationFactor and consistencyFactor is updated to 2.

Now, delete the CVR pvc-79ba260d-fad6-11e9-8791-42010a8000c5-cstor-sparse-pool-mch2 using kubectl delete command.

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #

Special notes for your reviewer:

Checklist:

• Fixes #
• Labelled this PR & related issue with documentation tag
• PR messages has document related information
• Labelled this PR & related issue with breaking-changes tag
• PR messages has breaking changes related information
• Labelled this PR & related issue with requires-upgrade tag
• PR messages has upgrade related information
• Commit has unit tests
• Commit has integration tests