Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement dynamic provisioning (beta) of PersistentVolumes via StorageClass #29006

Merged
merged 6 commits into from
Aug 18, 2016
+1,179 −348
Merged

Implement dynamic provisioning (beta) of PersistentVolumes via StorageClass #29006

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
6e4d95f
Dynamic provisioning V2 controller, provisioners, docs and tests.
jsafrane Aug 18, 2016
4b97db2
AWS changes for new provisioning model
jsafrane Aug 18, 2016
d942208
GCE changes for the new provisioning model
jsafrane Aug 18, 2016
f6fb99b
Provisioning documentation updates
jsafrane Aug 18, 2016
d8a95a3
Update matching logic with storage class
jsafrane Aug 18, 2016
bb5d562
Restore alpha behavior
jsafrane Aug 18, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 8 additions & 6 deletions cmd/kube-controller-manager/app/controllermanager.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -419,19 +419,21 @@ func StartControllers(s *options.CMServer, kubeClient *client.Client, kubeconfig
glog.Infof("Not starting %s apis", groupVersion)
}

provisioner, err := NewVolumeProvisioner(cloud, s.VolumeConfiguration)
alphaProvisioner, err := NewAlphaVolumeProvisioner(cloud, s.VolumeConfiguration)
if err != nil {
glog.Fatalf("A Provisioner could not be created: %v, but one was expected. Provisioning will not work. This functionality is considered an early Alpha version.", err)
glog.Fatalf("An backward-compatible provisioner could not be created: %v, but one was expected. Provisioning will not work. This functionality is considered an early Alpha version.", err)
}

volumeController := persistentvolumecontroller.NewPersistentVolumeController(
clientset.NewForConfigOrDie(restclient.AddUserAgent(kubeconfig, "persistent-volume-binder")),
s.PVClaimBinderSyncPeriod.Duration,
provisioner,
ProbeRecyclableVolumePlugins(s.VolumeConfiguration),
alphaProvisioner,
ProbeControllerVolumePlugins(cloud, s.VolumeConfiguration),
cloud,
s.ClusterName,
nil, nil, nil,
nil, // volumeSource
nil, // claimSource
nil, // classSource
nil, // eventRecorder
s.VolumeConfiguration.EnableDynamicProvisioning,
)
volumeController.Run()
Expand Down
39 changes: 27 additions & 12 deletions cmd/kube-controller-manager/app/plugins.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,8 +62,10 @@ func ProbeAttachableVolumePlugins(config componentconfig.VolumeConfiguration) []
return allPlugins
}

// ProbeRecyclableVolumePlugins collects all persistent volume plugins into an easy to use list.
func ProbeRecyclableVolumePlugins(config componentconfig.VolumeConfiguration) []volume.VolumePlugin {
// ProbeControllerVolumePlugins collects all persistent volume plugins into an
// easy to use list. Only volume plugins that implement any of
// provisioner/recycler/deleter interface should be returned.
func ProbeControllerVolumePlugins(cloud cloudprovider.Interface, config componentconfig.VolumeConfiguration) []volume.VolumePlugin {
allPlugins := []volume.VolumePlugin{}

// The list of plugins to probe is decided by this binary, not
Expand All @@ -79,6 +81,7 @@ func ProbeRecyclableVolumePlugins(config componentconfig.VolumeConfiguration) []
RecyclerMinimumTimeout: int(config.PersistentVolumeRecyclerConfiguration.MinimumTimeoutHostPath),
RecyclerTimeoutIncrement: int(config.PersistentVolumeRecyclerConfiguration.IncrementTimeoutHostPath),
RecyclerPodTemplate: volume.NewPersistentVolumeRecyclerPodTemplate(),
ProvisioningEnabled: config.EnableHostPathProvisioning,
}
if err := AttemptToLoadRecycler(config.PersistentVolumeRecyclerConfiguration.PodTemplateFilePathHostPath, &hostPathConfig); err != nil {
glog.Fatalf("Could not create hostpath recycler pod from file %s: %+v", config.PersistentVolumeRecyclerConfiguration.PodTemplateFilePathHostPath, err)
Expand All @@ -95,22 +98,34 @@ func ProbeRecyclableVolumePlugins(config componentconfig.VolumeConfiguration) []
}
allPlugins = append(allPlugins, nfs.ProbeVolumePlugins(nfsConfig)...)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At this point do we still need to open-code the list or can we iterate an "all plugins" list and find them dynamically?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This approach was turned down in PR #21552

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That was too magical. I meant in the other direction. We could have a
pkg/volume/all/all_plugins.go which explicitly contains a list of all
plugins and offers simple functions like all.ProbeVolumePlugins. This is
probably a different set of PRs to do this. My main gripe is that
controller manager is special-casing things like NFs, when it really should
be generic mechanism. Plugins should be more or less opaque. I'll file an
issue

On Mon, Aug 8, 2016 at 4:05 AM, Jan Šafránek notifications@github.com
wrote:

In cmd/kube-controller-manager/app/plugins.go
#29006 (comment)
:

@@ -95,32 +97,18 @@ func ProbeRecyclableVolumePlugins(config componentconfig.VolumeConfiguration) []
}
allPlugins = append(allPlugins, nfs.ProbeVolumePlugins(nfsConfig)...)

This approach was turned down in PR #21552
#21552

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/kubernetes/pull/29006/files/ec13593208a8c485eaded742cd20663864293c05#r73857700,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVETXYdApodtqcD-BBXfppNjNTXHAks5qdw1ygaJpZM4JNdmW
.


allPlugins = append(allPlugins, aws_ebs.ProbeVolumePlugins()...)
allPlugins = append(allPlugins, gce_pd.ProbeVolumePlugins()...)
allPlugins = append(allPlugins, cinder.ProbeVolumePlugins()...)
allPlugins = append(allPlugins, vsphere_volume.ProbeVolumePlugins()...)
if cloud != nil {
switch {
case aws.ProviderName == cloud.ProviderName():
allPlugins = append(allPlugins, aws_ebs.ProbeVolumePlugins()...)
case gce.ProviderName == cloud.ProviderName():
allPlugins = append(allPlugins, gce_pd.ProbeVolumePlugins()...)
case openstack.ProviderName == cloud.ProviderName():
allPlugins = append(allPlugins, cinder.ProbeVolumePlugins()...)
case vsphere.ProviderName == cloud.ProviderName():
allPlugins = append(allPlugins, vsphere_volume.ProbeVolumePlugins()...)
}
}

return allPlugins
}

// NewVolumeProvisioner returns a volume provisioner to use when running in a cloud or development environment.
// The beta implementation of provisioning allows 1 implied provisioner per cloud, until we allow configuration of many.
// We explicitly map clouds to volume plugins here which allows us to configure many later without backwards compatibility issues.
// Not all cloudproviders have provisioning capability, which is the reason for the bool in the return to tell the caller to expect one or not.
func NewVolumeProvisioner(cloud cloudprovider.Interface, config componentconfig.VolumeConfiguration) (volume.ProvisionableVolumePlugin, error) {
// NewAlphaVolumeProvisioner returns a volume provisioner to use when running in
// a cloud or development environment. The alpha implementation of provisioning
// allows 1 implied provisioner per cloud and is here only for compatibility
// with Kubernetes 1.3
// TODO: remove in Kubernetes 1.5
func NewAlphaVolumeProvisioner(cloud cloudprovider.Interface, config componentconfig.VolumeConfiguration) (volume.ProvisionableVolumePlugin, error) {
switch {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we really need this switch statement anymore?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at this closer the switch adds the volume type for a set cloud provider. I think we should leave this until we've tested scenario's like ebs running in non aws cloud provider.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd rather NOT pass cloud in at all. If someone tries to use EBS on GCE or vice-versa, it is not going to work, and I think that is OK.

case cloud == nil && config.EnableHostPathProvisioning:
return getProvisionablePluginFromVolumePlugins(host_path.ProbeVolumePlugins(volume.VolumeConfig{}))
return getProvisionablePluginFromVolumePlugins(host_path.ProbeVolumePlugins(
volume.VolumeConfig{
ProvisioningEnabled: true,
}))
case cloud != nil && aws.ProviderName == cloud.ProviderName():
return getProvisionablePluginFromVolumePlugins(aws_ebs.ProbeVolumePlugins())
case cloud != nil && gce.ProviderName == cloud.ProviderName():
Expand Down
16 changes: 8 additions & 8 deletions contrib/mesos/pkg/controllermanager/controllermanager.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -285,21 +285,21 @@ func (s *CMServer) Run(_ []string) error {
}
}

provisioner, err := kubecontrollermanager.NewVolumeProvisioner(cloud, s.VolumeConfiguration)
alphaProvisioner, err := kubecontrollermanager.NewAlphaVolumeProvisioner(cloud, s.VolumeConfiguration)
if err != nil {
glog.Fatalf("A Provisioner could not be created: %v, but one was expected. Provisioning will not work. This functionality is considered an early Alpha version.", err)
glog.Fatalf("An backward-compatible provisioner could not be created: %v, but one was expected. Provisioning will not work. This functionality is considered an early Alpha version.", err)
}

volumeController := persistentvolumecontroller.NewPersistentVolumeController(
clientset.NewForConfigOrDie(restclient.AddUserAgent(kubeconfig, "persistent-volume-binder")),
s.PVClaimBinderSyncPeriod.Duration,
provisioner,
kubecontrollermanager.ProbeRecyclableVolumePlugins(s.VolumeConfiguration),
alphaProvisioner,
kubecontrollermanager.ProbeControllerVolumePlugins(cloud, s.VolumeConfiguration),
cloud,
s.ClusterName,
nil,
nil,
nil,
nil, // volumeSource
nil, // claimSource
nil, // classSource
nil, // eventRecorder
s.VolumeConfiguration.EnableDynamicProvisioning,
)
volumeController.Run()
Expand Down
20 changes: 10 additions & 10 deletions docs/proposals/volume-provisioning.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -122,13 +122,13 @@ We propose that:
a match is found. The claim is `Pending` during this period.

4. With StorageClass instance, the controller finds volume plugin specified by
StorageClass.ProvisionerType.
StorageClass.Provisioner.

5. All provisioners are in-tree; they implement an interface called
`ProvisionableVolumePlugin`, which has a method called `NewProvisioner`
that returns a new provisioner.

6. The controller calls volume plugin `Provision` with ProvisionerParameters from the `StorageClass` configuration object.
6. The controller calls volume plugin `Provision` with Parameters from the `StorageClass` configuration object.

7. If `Provision` returns an error, the controller generates an event on the
claim and goes back to step 1., i.e. it will retry provisioning periodically
Expand Down Expand Up @@ -166,11 +166,11 @@ type StorageClass struct {
unversioned.TypeMeta `json:",inline"`
ObjectMeta `json:"metadata,omitempty"`

// ProvisionerType indicates the type of the provisioner.
ProvisionerType string `json:"provisionerType,omitempty"`
// Provisioner indicates the type of the provisioner.
Provisioner string `json:"provisioner,omitempty"`

// Parameters for dynamic volume provisioner.
ProvisionerParameters map[string]string `json:"provisionerParameters,omitempty"`
Parameters map[string]string `json:"parameters,omitempty"`
}

```
Expand Down Expand Up @@ -207,7 +207,7 @@ With the scheme outlined above the provisioner creates PVs using parameters spec
### Provisioner interface changes

`struct volume.VolumeOptions` (containing parameters for a provisioner plugin)
will be extended to contain StorageClass.ProvisionerParameters.
will be extended to contain StorageClass.Parameters.

The existing provisioner implementations will be modified to accept the StorageClass configuration object.

Expand All @@ -229,8 +229,8 @@ apiVersion: v1
kind: StorageClass
metadata:
name: aws-fast
provisionerType: kubernetes.io/aws-ebs
provisionerParameters:
provisioner: kubernetes.io/aws-ebs
parameters:
zone: us-east-1b
type: ssd
Expand All @@ -239,8 +239,8 @@ apiVersion: v1
kind: StorageClass
metadata:
name: aws-slow
provisionerType: kubernetes.io/aws-ebs
provisionerParameters:
provisioner: kubernetes.io/aws-ebs
parameters:
zone: us-east-1b
type: spinning
```
Expand Down
79 changes: 48 additions & 31 deletions examples/experimental/persistent-volume-provisioning/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,34 +43,49 @@ scripts that launch kube-controller-manager.

### Admin Configuration

No configuration is required by the admin! 3 cloud providers will be provided in the alpha version
of this feature: EBS, GCE, and Cinder.

When Kubernetes is running in one of those clouds, there will be an implied provisioner.
There is no provisioner when running outside of any of those 3 cloud providers.

A fourth provisioner is included for testing and development only. It creates HostPath volumes,
which will never work outside of a single node cluster. It is not supported in any way except for
local for testing and development. This provisioner may be used by passing
`--enable-hostpath-provisioner=true` to the controller manager while bringing it up.
When using the `hack/local_up_cluster.sh` script, this flag is turned off by default.
It may be turned on by setting the environment variable `ENABLE_HOSTPATH_PROVISIONER`
to true prior to running the script.

The admin must define `StorageClass` objects that describe named "classes" of storage offered in a cluster. Different classes might map to arbitrary levels or policies determined by the admin. When configuring a `StorageClass` object for persistent volume provisioning, the admin will need to describe the type of provisioner to use and the parameters that will be used by the provisioner when it provisions a `PersistentVolume` belonging to the class.

The name of a StorageClass object is significant, and is how users can request a particular class, by specifying the name in their `PersistentVolumeClaim`. The `provisioner` field must be specified as it determines what volume plugin is used for provisioning PVs. 2 cloud providers will be provided in the beta version of this feature: EBS and GCE. The `parameters` field contains the parameters that describe volumes belonging to the storage class. Different parameters may be accepted depending on the `provisioner`. For example, the value `io1`, for the parameter `type`, and the parameter `iopsPerGB` are specific to EBS . When a parameter is omitted, some default is used.

#### AWS

```yaml
kind: StorageClass
apiVersion: extensions/v1beta1
metadata:
name: slow
provisioner: kubernetes.io/aws-ebs
parameters:
type: io1
zone: us-east-1d
iopsPerGB: "10"
```
env ENABLE_HOSTPATH_PROVISIONER=true hack/local-up-cluster.sh
* `type`: `io1`, `gp2`, `sc1`, `st1`. See AWS docs for details. Default: `gp2`.
* `zone`: AWS zone. If not specified, a random zone in the same region as controller-manager will be chosen.
* `iopsPerGB`: only for `io1` volumes. I/O operations per second per GiB. AWS volume plugin multiplies this with size of requested volume to compute IOPS of the volume and caps it at 20 000 IOPS (maximum supported by AWS, see AWS docs).

#### GCE

```yaml
kind: StorageClass
apiVersion: extensions/v1beta1
metadata:
name: slow
provisioner: kubernetes.io/gce-pd
parameters:
type: pd-standard
zone: us-central1-a
```

* `type`: `pd-standard` or `pd-ssd`. Default: `pd-ssd`
* `zone`: GCE zone. If not specified, a random zone in the same region as controller-manager will be chosen.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is happening to the old annotation? Are we abandoning it? Can we allow the old annotation with the old behavior for 1 overlapping release and then remove it? We'll need a transition doc.


### User provisioning requests

Users request dynamically provisioned storage by including a storage class in their `PersistentVolumeClaim`.
The annotation `volume.alpha.kubernetes.io/storage-class` is used to access this experimental feature.
In the future, admins will be able to define many storage classes.
The storage class may remain in an annotation or become a field on the claim itself.

> The value of the storage-class annotation does not matter in the alpha version of this feature. There is
a single implied provisioner per cloud (which creates 1 kind of volume in the provider). The full version of the feature
will require that this value matches what is configured by the administrator.
The annotation `volume.beta.kubernetes.io/storage-class` is used to access this experimental feature. It is required that this value matches the name of a `StorageClass` configured by the administrator.
In the future, the storage class may remain in an annotation or become a field on the claim itself.

```
{
Expand All @@ -79,7 +94,7 @@ will require that this value matches what is configured by the administrator.
"metadata": {
"name": "claim1",
"annotations": {
"volume.alpha.kubernetes.io/storage-class": "foo"
"volume.beta.kubernetes.io/storage-class": "slow"
}
},
"spec": {
Expand All @@ -97,26 +112,28 @@ will require that this value matches what is configured by the administrator.
### Sample output
This example uses HostPath but any provisioner would follow the same flow.
This example uses GCE but any provisioner would follow the same flow.
First we note there are no Persistent Volumes in the cluster. After creating a claim, we see a new PV is created
First we note there are no Persistent Volumes in the cluster. After creating a storage class and a claim including that storage class, we see a new PV is created
and automatically bound to the claim requesting storage.
```
```
$ kubectl get pv

$ kubectl create -f examples/experimental/persistent-volume-provisioning/gce-pd.yaml
storageclass "slow" created

$ kubectl create -f examples/experimental/persistent-volume-provisioning/claim1.json
I1012 13:07:57.666759 22875 decoder.go:141] decoding stream as JSON
persistentvolumeclaim "claim1" created

$ kubectl get pv
NAME LABELS CAPACITY ACCESSMODES STATUS CLAIM REASON AGE
pv-hostpath-r6z5o createdby=hostpath-dynamic-provisioner 3Gi RWO Bound default/claim1 2s
NAME CAPACITY ACCESSMODES STATUS CLAIM REASON AGE
pvc-bb6d2f0c-534c-11e6-9348-42010af00002 3Gi RWO Bound default/claim1 4s

$ kubectl get pvc
NAME LABELS STATUS VOLUME CAPACITY ACCESSMODES AGE
claim1 <none> Bound pv-hostpath-r6z5o 3Gi RWO 7s
NAME LABELS STATUS VOLUME CAPACITY ACCESSMODES AGE
claim1 <none> Bound pvc-bb6d2f0c-534c-11e6-9348-42010af00002 3Gi RWO 7s

# delete the claim to release the volume
$ kubectl delete pvc claim1
Expand Down
9 changes: 9 additions & 0 deletions examples/experimental/persistent-volume-provisioning/aws-ebs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
kind: StorageClass
apiVersion: extensions/v1beta1
metadata:
name: slow
provisioner: kubernetes.io/aws-ebs
parameters:
type: io1
zone: us-east-1d
iopsPerGB: "10"
2 changes: 1 addition & 1 deletion examples/experimental/persistent-volume-provisioning/claim1.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
"metadata": {
"name": "claim1",
"annotations": {
"volume.alpha.kubernetes.io/storage-class": "foo"
"volume.beta.kubernetes.io/storage-class": "slow"
}
},
"spec": {
Expand Down
20 changes: 0 additions & 20 deletions examples/experimental/persistent-volume-provisioning/claim2.json
View file
Open in desktop

This file was deleted.

8 changes: 8 additions & 0 deletions examples/experimental/persistent-volume-provisioning/gce-pd.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
kind: StorageClass
apiVersion: extensions/v1beta1
metadata:
name: slow
provisioner: kubernetes.io/gce-pd
parameters:
type: pd-standard
zone: us-central1-a
Loading