feat(cvc-operator): add automatic scaling of volumereplicas for CSI volumes #1613

mittachaitu · 2020-02-11T14:01:14Z

Signed-off-by: mittachaitu sai.chaithanya@mayadata.io

What this PR does / why we need it:
This PR adds a feature to automatically ScaleUp/ScaleDown the volume replicas for CSI volumes.

Current CVC:

apiVersion: openebs.io/v1alpha1
kind: CStorVolumeClaim
metadata:
  annotations:
    openebs.io/volume-policy: ""
    openebs.io/volumeID: pvc-bd58d8d6-4d97-11ea-80a1-42010a8000d3
  creationTimestamp: "2020-02-12T13:01:10Z"
  finalizers:
  - cvc.openebs.io/finalizer
  generation: 2
  labels:
    openebs.io/cstor-pool-cluster: cstor-sparse-cspc
    openebs.io/pod-disruption-budget: cstor-sparse-cspcmssgq
  name: pvc-bd58d8d6-4d97-11ea-80a1-42010a8000d3
  namespace: openebs
  resourceVersion: "23259649"
  selfLink: /apis/openebs.io/v1alpha1/namespaces/openebs/cstorvolumeclaims/pvc-bd58d8d6-4d97-11ea-80a1-42010a8000d3
  uid: bd611b24-4d97-11ea-80a1-42010a8000d3
publish:
  nodeId: gke-sai-test-cluster-pool-1-2ad4a0b7-f2sd
spec:
  capacity:
    storage: 5Gi
  cstorVolumeRef:
    apiVersion: openebs.io/v1alpha1
    kind: CStorVolume
    name: pvc-bd58d8d6-4d97-11ea-80a1-42010a8000d3
    namespace: openebs
    resourceVersion: "23259625"
    uid: bd74b9ef-4d97-11ea-80a1-42010a8000d3
  policy:
    provision:
      replicaAffinity: false
    replica:
      affinity: null
      zvolWorkers: ""
    replicaPoolInfo:
      - poolName: cstor-sparse-cspc-l2ht
      - poolName: cstor-sparse-cspc-wgwf
      - poolName: cstor-sparse-cspc-wzt4
    target: {}
  replicaCount: 3
status:
  capacity:
    storage: 5Gi
  phase: Bound
  poolInfo:
  - cstor-sparse-cspc-l2ht
  - cstor-sparse-cspc-wgwf
  - cstor-sparse-cspc-wzt4
versionDetails:
  autoUpgrade: false
  desired: 1.7.0
  status:
    current: 1.7.0
    dependentsUpgraded: true
    lastUpdateTime: null
    state: ""

After creating PVC above CVC will be created in openebs namespace.

How the user/admin can perform scaling-up volume replicas:
In above CVC under cvc.spec.policy.replicaPool.poolInfo will show the volumereplicas pool information. Now to perform volume replica ScaleUp just add the pool entry under spec of CVC i.e spec.policy.replicaPool.poolInfo.
i.e Just change from

spec:
    replicaPoolInfo:
      - poolName: cstor-sparse-cspc-l2ht
      - poolName: cstor-sparse-cspc-wgwf
      - poolName: cstor-sparse-cspc-wzt4

to

spec:
    replicaPoolInfo:
      - poolName: cstor-sparse-cspc-l2ht
      - poolName: cstor-sparse-cspc-wgwf
      - poolName: cstor-sparse-cspc-wzt4
      - poolName: cstor-sparse-cspc-g54z

In above changes added the pool entry cstor-sparse-cspc-g54z. Rest of the things will be taken care by CVC controller.

How the user/admin can perform scaling-down volume replicas:
Same as above, to perform volume replica ScaleDown just remove the pool entry under spec of CVC i.e spec.policy.replicaPool.poolInfo.

spec:
    replicaPoolInfo:
      - poolName: cstor-sparse-cspc-l2ht
      - poolName: cstor-sparse-cspc-wgwf
      - poolName: cstor-sparse-cspc-wzt4
      - poolName: cstor-sparse-cspc-g54z

to

      ReplicaPoolInfo:
      - poolName: cstor-sparse-cspc-l2ht
      - poolName: cstor-sparse-cspc-wzt4
      - poolName: cstor-sparse-cspc-g54z

In above changes removed the pool entry cstor-sparse-cspc-wgwf. Rest of the things will be taken care by CVC controller.

How User/Admin can know the completed process:
User/Admin can verify from the CVC events (or) Status of CVC will show the current volume replica pool names i.e cvc.Status.PoolInfo.
How events will look like

Events:
  Type     Reason                 Age                From                         Message
  ----     ------                 ----               ----                         -------
  Normal   Synced                 32m                cstorvolumeclaim-controller  cstorvolumeclaim created successfully
  Warning  ScalingVolumeReplicas  20m (x2 over 21m)  cstorvolumeclaim-controller  scaling replicas from 3 to 4 in progress
  Normal   ScalingVolumeReplicas  20m                cstorvolumeclaim-controller  successfully scaled volume replicas to 4
  Warning  ScalingVolumeReplicas  15m                cstorvolumeclaim-controller  Scaling down volume replicas from 4 to 3 is in progress
  Normal   ScalingVolumeReplicas  14m                cstorvolumeclaim-controller  successfully scaled volume replicas to 3

How the user/admin can perform the migration of volume replicas from one pool to another pool:

Perform the scaledown step using CVC(i.e remove the pool entry under spec of CVC).
Perform the scaleup step using CVC(i.e Add the required pool entry under spec of CVC).

How Non-CSI volumes will scaleup/scaledown volume replicas:
ScaleUp and ScaleDown links.

Note:

This PR handles the PDB modifications according to scale up and scale down of volume replicas.
After performing scaling up/down the replica count will not be updated on CVC(treating it as initial replica count).
Scaling Down more than one replica at a time is not supported.

Release Notes:
Add support for scaling volume replicas for CSI Volumes via CVC(Single point edit to perform scaling operation).

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #

Special notes for your reviewer:

Workflow discussion doc workflow.
Integration testing will be raised in different PR.
Volume Replica ScaleUp PRs chore: (cstor-volume-mgmt) Add support to update cstorvolume status with known replicas #1447 and fix(cvc, volume creation): populate desired replication factor during volume creation #1462
Volume Replica ScaleDown PR in cStorVolumeMgmt component.
This PR handles the PDB modifications according to scale up and scale down of volume replicas.

Checklist:

Fixes #
Labelled this PR & related issue with documentation tag
PR messages has document related information
Labelled this PR & related issue with breaking-changes tag
PR messages has breaking changes related information
Labelled this PR & related issue with requires-upgrade tag
PR messages has upgrade related information
Commit has unit tests
Commit has integration tests

Signed-off-by: mittachaitu <sai.chaithanya@mayadata.io>

cmd/cvc-operator/controller/cstorvolumeclaim.go

cmd/cvc-operator/controller/cvc_controller.go

cmd/cvc-operator/controller/cstorvolumeclaim.go

cmd/cvc-operator/controller/cvc_controller.go

Signed-off-by: mittachaitu <sai.chaithanya@mayadata.io>

cmd/cvc-operator/controller/cstorvolumeclaim.go

Signed-off-by: mittachaitu <sai.chaithanya@mayadata.io>

cmd/cvc-operator/controller/cstorvolumeclaim.go

Signed-off-by: mittachaitu <sai.chaithanya@mayadata.io>

cmd/cvc-operator/controller/cstorvolumeclaim.go

vishnuitta · 2020-02-20T09:35:06Z

cmd/cvc-operator/controller/cstorvolumeclaim.go

@@ -375,7 +384,10 @@ func (c *CVCController) distributeCVRs(
 	for count, pool := range usablePoolList.Items {


poolList, err := listCStorPools(cspcName, claim.Spec.ReplicaCount)
Just list the cstor pools in listCStorPools, and have the check on count after getting usable pool list against pendingReplicaCount rather than spec.ReplicaCount

changing name of ReplicaCount?

'if else' code at if claim.Spec.CstorVolumeSource != "" { need to be rewritten

added the check other two comments will be fixed while moving the code to cstor-operators repo.

vishnuitta · 2020-02-20T10:29:05Z

cmd/cvc-operator/controller/cvc_controller.go

@@ -297,6 +313,8 @@ func (c *CVCController) createVolumeOperation(cvc *apis.CStorVolumeClaim) (*apis
 	cvc.Spec.Policy = volumePolicy.Spec
 	cvc.Status.Phase = apis.CStorVolumeClaimPhaseBound
 	cvc.Status.Capacity = cvc.Spec.Capacity
+	// update volume replica pool information on cvc spec and status
+	addReplicaPoolInfo(cvc, poolNames)


how about changing it as receiver function like cvc.addReplicaPoolInfo(poolNames)

cmd/cvc-operator/controller/cvc_controller.go

vishnuitta · 2020-02-20T11:01:45Z

cmd/cvc-operator/controller/cstorvolumeclaim.go

+// names and current replica pool names
+// 1. Below function will check whether there is any change in desired replica
+//    pool names and current replica pool names.
+func (c *CVCController) isCVCScalePending(cvc *apis.CStorVolumeClaim) bool {


add a check that cvc is in bound state.. if not, return false

and also add comment that scale up/down of cvc will not work until cvc is in bound state

Not required to add a check. Resize, scale Up/Down will execute only once CVC is bound.

vishnuitta · 2020-02-20T12:29:52Z

cmd/cvc-operator/controller/cvc_controller.go

+	} else if len(cvc.Spec.Policy.ReplicaPool.PoolInfo) < len(cvc.Status.PoolInfo) {
+		err = scaleDownVolumeReplicas(cvc)
+	} else {
+		c.recorder.Event(cvc, corev1.EventTypeWarning, "Migration",


should this limitation get into webhooks?

Once the webhook for CVC is ready we will have following checks

Repetition of pool names either under spec or status shouldn't be there.

InitialReplicaCount shouldn't be modified.

The existing pool name can't be updated with a new pool name(Which is migration case).

Not allow scale down/up of volume replicas when one other in progress.

Not allow more that one replica at a time for scale down.

Failure of scale up/down if added pool is not healthy ? ( but admin/user shouldn't be providing the unhealthy pool name in first place)

vishnuitta

lets make replicaPool as replicaPoolInfo

Signed-off-by: mittachaitu <sai.chaithanya@mayadata.io>

cmd/cvc-operator/controller/cstorvolumeclaim.go

vishnuitta · 2020-02-24T17:28:09Z

cmd/cvc-operator/controller/cstorvolumeclaim.go

+		return errors.Wrapf(err, "failed to get cstorvolumes object %s", cvc.Name)
+	}
+	if cvObj.Spec.DesiredReplicationFactor < drCount {
+		cvObj.Spec.DesiredReplicationFactor = drCount


should we consider picking replicationCount from CVC instead of CV for cstor-volume-mgmt?
This will avoid maintaining this info at multiple places. So, cstor-volume-mgmt will do like:

pick the cvc.spec.ReplicaCount if len of ReplicaPoolInfo is zero

else consider replicationFactor as len of ReplicaPoolInfo if it is present
what steps does cstor-vol-mgmt performs on noticing change in DRF?

I tried to find from code, but unable to find other than scale down

this info on how cstor-vol-mgmt works helps in review

I will update PR with relevant pr's

https://github.com/openebs/maya/blob/dd6152021192d2bebcd77487da0b7bdef7249dfe/cmd/cstor-volume-mgmt/controller/volume-controller/handler.go#L483

Comments will explain the case

https://github.com/openebs/maya/blob/dd6152021192d2bebcd77487da0b7bdef7249dfe/cmd/cstor-volume-mgmt/controller/volume-controller/handler.go#L534

Scaleup function in cstor-volume-mgmt and updated the PR description with relevant PR's.

cmd/cvc-operator/controller/cstorvolumeclaim.go

cmd/cvc-operator/controller/cvc_controller.go

prateekpandey14 · 2020-02-25T09:56:48Z

cmd/cvc-operator/controller/cvc_controller.go

+	} else if len(cvc.Spec.Policy.ReplicaPool.PoolInfo) < len(cvc.Status.PoolInfo) {
+		err = scaleDownVolumeReplicas(cvc)
+	} else {
+		c.recorder.Event(cvc, corev1.EventTypeWarning, "Migration",


Failure of scale up/down if added pool is not healthy ? ( but admin/user shouldn't be providing the unhealthy pool name in first place)

prateekpandey14 · 2020-02-25T10:15:57Z

cmd/cvc-operator/controller/cstorvolumeclaim.go

+	// If more than one replica was scale down at a time keep on return the error
+	if (cvObj.Spec.ReplicationFactor - drCount) > 1 {
+		return errors.Wrapf(err,
+			"cann't perform %d replicas scaledown at a time",


("replica scale-down can not be performed more then 1 at a time, requested for %d", (cvObj.Spec.DesiredReplicationFactor - drCount) )

Don't Wrapf the error here , it will nil , use errors.Errorf() instead.

prateekpandey14 · 2020-02-25T10:43:45Z

cmd/cvc-operator/controller/cstorvolumeclaim.go

+	}
+	cvrObj, err = getScaleDownCVR(cvc)
+	if err != nil && !k8serror.IsNotFound(err) {
+		return errors.Wrapf(err, "failed to get scale down CVR object")


can we have error like --> ("failed to get cvr %d , requested for scale down operation", cvc.Name)

prateekpandey14 · 2020-02-25T13:34:09Z

cmd/cvc-operator/controller/cstorvolumeclaim.go

+//        created PDB name.
+// 2. Update CVC label to point it to newely PDB got from above step and also
+//    replicas pool information on status of CVC.
+func handlePostScalingProcess(cvc *apis.CStorVolumeClaim,


nit: func handlePostScalingOperations() ?

or use func updatePDBForScaledVolume () ?

vishnuitta · 2020-02-25T14:18:36Z

cmd/cvc-operator/controller/cstorvolumeclaim.go

+	}
+	for _, poolName := range newPoolNames {
+		if _, ok := replicaPoolMap[poolName]; ok {
+			scaledRPNames = append(scaledRPNames, poolName)


just return error as 'in-progress' if !ok and update the status when all the required CVRs are required

vishnuitta · 2020-02-25T14:22:23Z

cmd/cvc-operator/controller/cstorvolumeclaim.go

+//         HAVolume) and update the replica pool info on CVC(API calls).
+//    2.2: If scalingUp was going then return error saying scalingUp was in
+//      progress.
+func verifyAndUpdateScaleUpInfo(cvc *apis.CStorVolumeClaim, cvObj *apis.CStorVolume) error {


change the function name as updateCVCWithScaledUpInfo

modifyCVCWithScaledUpInfo

vishnuitta · 2020-02-25T14:23:11Z

cmd/cvc-operator/controller/cstorvolumeclaim.go

+			klog.Errorf("%s", errorMsg)
+			continue
+		}
+		hash, err := hash.Hash(pvName + "-" + poolName)


it makes sense to follow same key for hash..

vishnuitta · 2020-02-25T14:49:38Z

cmd/cvc-operator/controller/cstorvolumeclaim.go

+//        created PDB name.
+// 2. Update CVC label to point it to newely PDB got from above step and also
+//    replicas pool information on status of CVC.
+func handlePostScalingProcess(cvc *apis.CStorVolumeClaim,


change name to updatePDBForVolume

vishnuitta · 2020-02-25T14:51:05Z

cmd/cvc-operator/controller/cvc_controller.go

+	c.recorder.Eventf(cvc,
+		corev1.EventTypeNormal,
+		"ScalingVolumeReplicas",
+		"successfully scaled volume replicas to %d", len(cvc.Status.PoolInfo))


you need to make cvc point to new object

vishnuitta · 2020-02-25T14:58:13Z

cmd/cvc-operator/controller/cstorvolumeclaim.go

+		return errors.Wrapf(err, "failed to get cstorvolumes object %s", cvc.Name)
+	}
+	// If more than one replica was scale down at a time keep on return the error
+	if (cvObj.Spec.ReplicationFactor - drCount) > 1 {


this can get into webhook

we need similar ones for scale up also

~~don't allow scale down if DRF != RF~~

check for spec's and status's ReplicaInfo in webhook

vishnuitta · 2020-02-25T16:08:03Z

cmd/cvc-operator/controller/cstorvolumeclaim.go

+	var err error
+	cvcCopy := cvc.DeepCopy()
+	cvc.Status.PoolInfo = []string{}
+	cvc.Status.PoolInfo = append(cvc.Status.PoolInfo, currentRPNames...)


you can remove it and let caller take care of this

Signed-off-by: mittachaitu <sai.chaithanya@mayadata.io>

vishnuitta · 2020-02-25T17:42:25Z

cmd/cvc-operator/controller/cstorvolumeclaim.go

-	pdbName, hasPDB := cvcObj.GetLabels()[string(apis.PodDisruptionBudgetKey)]
-	pdbLabels := cvclaim.GetPDBPoolLabels(cvcObj.Status.PoolInfo)
-	labelSelector := apispdb.GetPDBLabelSelector(pdbLabels)
+func getUpdatePDBForVolume(cvcObj *apis.CStorVolumeClaim) (string, error) {


getModifiedPDBForScaledVolume

vishnuitta · 2020-02-25T17:42:48Z

cmd/cvc-operator/controller/cstorvolumeclaim.go

-	pdbName, hasPDB := cvcObj.GetLabels()[string(apis.PodDisruptionBudgetKey)]
-	pdbLabels := cvclaim.GetPDBPoolLabels(cvcObj.Status.PoolInfo)
-	labelSelector := apispdb.GetPDBLabelSelector(pdbLabels)
+func getUpdatePDBForVolume(cvcObj *apis.CStorVolumeClaim) (string, error) {


getModifiedPDBForScaledVolume

cmd/cvc-operator/controller/cstorvolumeclaim.go

vishnuitta

changes are good

Signed-off-by: mittachaitu <sai.chaithanya@mayadata.io>

cmd/cvc-operator/controller/cstorvolumeclaim.go

prateekpandey14

minor nit pick.
/lgtm

Signed-off-by: mittachaitu <sai.chaithanya@mayadata.io>

feat(cvc-operator): add automatic scaling of volumereplicas via CVC

af522b4

Signed-off-by: mittachaitu <sai.chaithanya@mayadata.io>

golangcibot reviewed Feb 11, 2020

View reviewed changes

This commit fixes the bugs in replica scaling feature

21245cd

Signed-off-by: mittachaitu <sai.chaithanya@mayadata.io>

mittachaitu changed the title ~~[WIP] feat(cvc-operator): add automatic scaling of volumereplicas via CVC~~ [WIP] feat(cvc-operator): add automatic scaling of volumereplicas for CSI volumes Feb 12, 2020

golangcibot reviewed Feb 12, 2020

View reviewed changes

cmd/cvc-operator/controller/cstorvolumeclaim.go Outdated Show resolved Hide resolved

cmd/cvc-operator/controller/cstorvolumeclaim.go Show resolved Hide resolved

cmd/cvc-operator/controller/cstorvolumeclaim.go Outdated Show resolved Hide resolved

mittachaitu changed the title ~~[WIP] feat(cvc-operator): add automatic scaling of volumereplicas for CSI volumes~~ feat(cvc-operator): add automatic scaling of volumereplicas for CSI volumes Feb 12, 2020

This commit fixes the golang ci linting issues

641bc8c

Signed-off-by: mittachaitu <sai.chaithanya@mayadata.io>

mittachaitu requested review from prateekpandey14, payes, shubham14bajpai and sonasingh46 and removed request for payes February 12, 2020 14:24

This commit adds autogenerated file

0030e9a

Signed-off-by: mittachaitu <sai.chaithanya@mayadata.io>

mittachaitu requested a review from payes February 12, 2020 14:34

This commit updates getNamespace() to openebsNamespace variable

15cb229

Signed-off-by: mittachaitu <sai.chaithanya@mayadata.io>

mittachaitu commented Feb 13, 2020

View reviewed changes

cmd/cvc-operator/controller/cstorvolumeclaim.go Outdated Show resolved Hide resolved

This commit fixes the issues in travis

2129e1e

Signed-off-by: mittachaitu <sai.chaithanya@mayadata.io>

mittachaitu mentioned this pull request Feb 17, 2020

(WIP)feat(BDD): add positive BDD to perform CStorVolumeReplica ScaleUp/ScaleDown #1617

Closed

9 tasks

mittachaitu self-assigned this Feb 20, 2020

vishnuitta reviewed Feb 20, 2020

View reviewed changes

cmd/cvc-operator/controller/cstorvolumeclaim.go Show resolved Hide resolved

vishnuitta reviewed Feb 20, 2020

View reviewed changes

cmd/cvc-operator/controller/cstorvolumeclaim.go Outdated Show resolved Hide resolved

vishnuitta reviewed Feb 20, 2020

View reviewed changes

cmd/cvc-operator/controller/cvc_controller.go Show resolved Hide resolved

vishnuitta reviewed Feb 20, 2020

View reviewed changes

This commit address the review comments

16f4afa

Signed-off-by: mittachaitu <sai.chaithanya@mayadata.io>

vishnuitta reviewed Feb 24, 2020

View reviewed changes

cmd/cvc-operator/controller/cstorvolumeclaim.go Show resolved Hide resolved

vishnuitta reviewed Feb 24, 2020

View reviewed changes

cmd/cvc-operator/controller/cstorvolumeclaim.go Outdated Show resolved Hide resolved

prateekpandey14 reviewed Feb 25, 2020

View reviewed changes

cmd/cvc-operator/controller/cvc_controller.go Outdated Show resolved Hide resolved

prateekpandey14 reviewed Feb 25, 2020

View reviewed changes

vishnuitta reviewed Feb 25, 2020

View reviewed changes

mittachaitu added 2 commits February 25, 2020 21:54

This commit incorporates review comments

715acc8

Signed-off-by: mittachaitu <sai.chaithanya@mayadata.io>

This commit handles PDB creation

68c2eae

Signed-off-by: mittachaitu <sai.chaithanya@mayadata.io>

vishnuitta reviewed Feb 25, 2020

View reviewed changes

cmd/cvc-operator/controller/cstorvolumeclaim.go Outdated Show resolved Hide resolved

vishnuitta approved these changes Feb 25, 2020

View reviewed changes

This commit fixes the function signature

ecf5801

Signed-off-by: mittachaitu <sai.chaithanya@mayadata.io>

prateekpandey14 reviewed Feb 26, 2020

View reviewed changes

cmd/cvc-operator/controller/cstorvolumeclaim.go Outdated Show resolved Hide resolved

prateekpandey14 approved these changes Feb 26, 2020

View reviewed changes

This commit fixes typo in logs

05b4bf2

Signed-off-by: mittachaitu <sai.chaithanya@mayadata.io>

prateekpandey14 merged commit 26fcac5 into openebs-archive:master Feb 26, 2020

mittachaitu mentioned this pull request Feb 27, 2020

feat(validation): add webhook validations for CVC replica scale #1621

Merged

9 tasks

mittachaitu added the pr/release-note PR should be included in release notes label Mar 11, 2020

mittachaitu mentioned this pull request Mar 17, 2020

Migration of volume replicas from one pool to another pool mayadata-io/cstorpoolauto#100

Open

		@@ -375,7 +384,10 @@ func (c *CVCController) distributeCVRs(
		for count, pool := range usablePoolList.Items {

feat(cvc-operator): add automatic scaling of volumereplicas for CSI volumes #1613

feat(cvc-operator): add automatic scaling of volumereplicas for CSI volumes #1613

Conversation

mittachaitu commented Feb 11, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mittachaitu Feb 20, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vishnuitta left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mittachaitu Feb 25, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vishnuitta Feb 25, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vishnuitta left a comment

Choose a reason for hiding this comment

prateekpandey14 left a comment

Choose a reason for hiding this comment

mittachaitu commented Feb 11, 2020 •

edited

Loading

mittachaitu Feb 20, 2020 •

edited

Loading

mittachaitu Feb 25, 2020 •

edited

Loading

vishnuitta Feb 25, 2020 •

edited

Loading