@zhan849 Can you explain more on it wrt to the support required and changes in API?

@johnugeorge I'm thinking about the following:

1. Add a slice of volume claim templates in ReplicaSpec

VolumeClaimTemplates []corev1.PersistentVolumeClaim `json:"volumeClaimTemplates,omitempty"`

2. In ReplicaSpec.Spec.InitContainers and ReplicaSpec.Spec.Containers, if any of the container requests a volume whose name is NOT in ReplicaSpec.Spec.Volumes but in ReplicaSpec.VolumeClaimTemplates, we create PersistentVolumeClaim objects for that container. Since containers are named with "-i" suffix, we can name the PVCs similarly.

3. When the managing object (i.e TFJob or PyTorchJob) is deleted, we delete the volumes as well

I can send out the PR for the community to review if the general direction is agreed.

/cc @richardsliu
/cc @gaocegege

The idea SGTM while I think storage class already supports dynamic volume provision, what things should we do in the use case?

@gaocegege yes storage class should define how the volume should actually be provisioned. We already started some experiments in a forked branch, do you guys want me to submit a brief proposal or you want to implement it. it'd be something similar to stateful set

@zhan849 I am glad to see your proposal! Thanks for your contribution.

@richardsliu @zhan849

Should we place the feature in common-operator? It is general for all PS-worker based training jobs

@gaocegege Yes I think so. So it should be in the next API version.

@gaocegege @richardsliu agreed. will close this one and open new one under common-operator.

And also for design doc, given all the discussion threads under tf-operator, I'd suggest that we keep it there and check into community repo after the PR is finalized and merged