I assume this means things like NFS for which provisioning is not just an API call and is not consistent across installations. But how can you make it possible? Something like a PVSourceOverride that calls a custom script? Maybe that's version 3.0

I am not sure if this is what @markturansky intends but I think implementations of PersistentVolumeSource should be independent of the VolumePlugin implementations where it makes sense. For GCE PD for example they are one and the same, but for NFS there can be multiple PersistentVolumeSource implementors which result in NFS exports. There can also be a PersistentVolumeSource which can produce both NFS and iSCSI exports for example.
In the case of just plain NFS with no cloud provider a custom PVSource would make sense.

So long as a plugin can call exec and the admin creating the PVControllers can specify what to call, I suppose the caveat should be "some assembly required" for some types of volumes.

@smarterclayton Having to make many ReplicationControllers with Replicas=1 and 1 claim is a bad user experience.

if a user were to lay claim to a PersistentVolumeController -- which is a creator and manager of its own pool of storage -- could we use this in conjunction with a ReplicationController to scale something like MongoDB? Different dev effort than this PR, but this PR's work could be steps towards solving that scaling issue.

On Jul 14, 2015, at 2:45 PM, Mark Turansky notifications@github.com wrote:

In docs/design/persistent-volume-provisioning.md
#6773 (comment)
:

@@ -0,0 +1,122 @@
+# Persistent Volume Provisioning
+
+This document proposes a model for dynamically provisioning Persistent Volumes
+
+### tl;dr
+
+One new API kinds:
+
+A PersistentVolumeController (PVC) is a storage resource provisioned by an administrator. PVCs do not have a namespace. Like a ReplicationController maintains a minimum number of "replicas" of a pod in the system, so a PersistentVolumeController maintains a minimum number of "replicas" of a certain type/size of PersistentVolume available in the system.
+

@smarterclayton https://github.com/smarterclayton Having to make many
ReplicationControllers with Replicas=1 and 1 claim is a bad user experience.

if a user were to lay claim to a PersistentVolumeController -- which is a
creator and manager of its own pool of storage -- could we use this in
conjunction with a ReplicationController to scale something like MongoDB?
Different dev effort than this PR, but this PR's work could be steps
towards solving that scaling issue.

Possible. The design of how we assign unique volumes to pods under a
replication controller (and reuse them when pods dies) is really important,
so we should try and get the discussion moving forward on that and reach
closure. Nominal services are tied to this.

—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/pull/6773/files#r34603920
.

How does the system allow for this? My question about this design proposal is that it seems to imply that the easiest way to provision volume types is to write Go code that calls GCE apis. That works well specifically for GCE/AWS/Cinder (in the normal case) but does not work very well for any organization that has to script their storage creation (by using ansible or some other tool to provision new machines or carve up NFS mounts). It also doesn't work very well for users with custom needs on GCE, or the ability to dynamically react to a claim and create a volume on demand. This seems like a hard problem to generically solve (provisioning storage of many different types) such that I'd ask why it has to be implemented in a formal pattern, vs by an integration that someone writes to watch for new claims or a lack of volumes and go create some more.

I'd like to be able to run a pod on the server hosting the volumes (assuming it is itself a node in the cluster) in order to create a volume.

Why does it have to be on the server hosting the volumes?

On Thu, Aug 20, 2015 at 3:30 PM, Paul Morie notifications@github.com
wrote:

In docs/design/persistent-volume-provisioning.md
#6773 (comment):

+PersistentVolumeControllerManager is a singleton control loop running in master that manages all PVControllers in the system. The PVCM reconciles the current supply of available PersistentVolumes in the system with the desired levels according to the PVControllers. This process is similar to the ReplicationManager that manages ReplicationControllers.
+
+Three new volume plugin interfaces:
+
+* Recycler -- knows how to scrub a volume clean so it can become available again as a resource
+* Creator -- create new instances of a PV from a template.
+* Deleter -- deletes instances of a PV and allows the plugin to determine how to remove it from the underlying infrastructure
+
+Volume plugins can implement any applicable interfaces. Each plugin will document its own support for dynamic provisioning.
+
+
+### Goals
+
+* Allow administrators to describe minimum and maximum storage levels comprised of many kinds of PersistentVolumes
+* Allow the dynamic creation and reclamation of persistent volumes (to the fullest extent for each type of storage provider)

I'd like to be able to run a pod on the server hosting the volumes
(assuming it is itself a node in the cluster) in order to create a volume.

—
Reply to this email directly or view it on GitHub
https://github.com/kubernetes/kubernetes/pull/6773/files#r37571168.

Clayton Coleman | Lead Engineer, OpenShift

@smarterclayton yes, this works much more easily for volumes w/ APIs.

I was thinking about making the strategy for creation be plugin-based. Perhaps the strategies are compiled in the /plugins package and you choose one by name via config (with a sensible default, of course).

If we made creation pluggable, @pmorie can run his pod to create a volume or Go code can call a provider's API.

@smarterclayton Are you arguing that this doesn't need API objects, but could just be a super-privileged pod that knows policy and watches across all namespaces for unfulfilled claims and does whatever it needs to do to make new PVs?

This whole design-doc sort of parallels the work @bprashanth is doing on load-balancers, so I'd like to keep them similar in form.

@saad-ali also

I think the key distinction is that the in tree controller would only
support a few obviously core types (like a cloud provider). A custom
controller would not be in tree, or compiled with the manager, and we would
disable it.

On Thu, Aug 20, 2015 at 8:46 PM, Mark Turansky notifications@github.com
wrote:

In docs/design/persistent-volume-provisioning.md
#6773 (comment):

+PersistentVolumeControllerManager is a singleton control loop running in master that manages all PVControllers in the system. The PVCM reconciles the current supply of available PersistentVolumes in the system with the desired levels according to the PVControllers. This process is similar to the ReplicationManager that manages ReplicationControllers.
+
+Three new volume plugin interfaces:
+
+* Recycler -- knows how to scrub a volume clean so it can become available again as a resource
+* Creator -- create new instances of a PV from a template.
+* Deleter -- deletes instances of a PV and allows the plugin to determine how to remove it from the underlying infrastructure
+
+Volume plugins can implement any applicable interfaces. Each plugin will document its own support for dynamic provisioning.
+
+
+### Goals
+
+* Allow administrators to describe minimum and maximum storage levels comprised of many kinds of PersistentVolumes
+* Allow the dynamic creation and reclamation of persistent volumes (to the fullest extent for each type of storage provider)

@smarterclayton https://github.com/smarterclayton yes, this works much
more easily for volumes w/ APIs.

I was thinking about making the strategy for creation be plugin-based.
Perhaps the strategies are compiled in the /plugins package and you choose
one by name via config (with a sensible default, of course).

If we made creation pluggable, @pmorie https://github.com/pmorie can
run his pod to create a volume or Go code can call a provider's API.

—
Reply to this email directly or view it on GitHub
https://github.com/kubernetes/kubernetes/pull/6773/files#r37597422.

Clayton Coleman | Lead Engineer, OpenShift

@thockin I'm arguing exactly what you said here: https://github.com/kubernetes/kubernetes/pull/6773/files#r37607582

PVs are analogous to Nodes. We don't have NodeSets. We don't have CloudLoadBalancerSets. We shouldn't need PersistentVolumeSets. Create PVs on demand in response to PVCs. One could also potentially horizontally auto-scale to keep a small amount of burst capacity.

Devil's advocate - will we ever have some configuration for a k8s cluster that defines the minimum number of nodes, the maximum number of nodes, the distribution of node sizes for auto-creation, the threshold for auto-creation/deletion, and hysteresis? We actually DO have this config but it isn't stored in kubernetes - it's in the cloud auto-scaler.

A PVSet (bad name) as proposed here is the analog to that. We could implement this as a pod with no state in the API, but the state has to go SOMEWHERE. We could argue that this should be a config object, once that exists and in the mean time just expect cmdline flags on a pod (or secrets :)

And then there is network ingress. I think ingress follows the same pattern. @bprashanth

If we do ever add such types, we should involve the auto-scaling team.

There's a desire to move from "active" names to "passive" names in the API. PersistentVolumePolicy? PVPool? PVSet?

I think @bgrant0607 wants to rename replicationController -> replicaSet eventually..

I'm not sure I buy the last sentence.

I can change to PersistentVolumeSet if that's where RC is headed. This thing makes a set of PVs.

I think I meant "PVs" in that last sentence, but that sentence also provides 0 value to the design doc because it may not be always true. I'll remove it.

Observation, before having read the whole proposal:

PersistentVolumes are like Nodes -- they represent infrastructure, either provisioned by an admin on bare metal or potentially horizontally auto-scaled or provisioned on demand on a public/private cloud. It doesn't seem like there will be a uniform implementation across cloud providers. We don't have NodeSets partly for this reason.

PersistentVolumeClaims are analogous to Pods. We need replication of PVCs, but not independently of the Pods that consume them. That's discussed in #260.

Can you explain the objection to PersistentVolumeSet? If they're all provisioned from the same template, it seems like the name correctly parallels ReplicaSet.

Also, I didn't understand the comment about PVCs. IIUC they're proposing replication/scaling of PersistentVolumes here, not replication of PVCs.