@dims: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dims

The full list of commands accepted by this bot can be found here.

The pull request process is described here

/assign @gnufied @xing-yang

/hold

cc @jsafrane @msau42

I think let's not rush to turn this on by GA this release. Firstly, let's wait for #98243 to get merged.

Instead, we can target to turn it on by default and remain it as beta in this release. At the same time, there is a list of criteria that each driver should follow to get it to GA. Let's check all the requirement has been met and then do the flip change.

@Jiawei0227 the openstack csi driver and the external cloud provider has been stable and working for a looooong time now. i just don't understand why we are dragging this.

https://github.com/kubernetes/kubernetes/pulls?q=is%3Apr+author%3Adims+openstack+in%3Asubject

@dims @jsafrane has started doing testing of openstack csi migration and has already found bugs. We're putting together a list of concrete testing criteria to ensure that we have adequate coverage and confidence in the feature.

@jsafrane would you like to take this over? ( cc @gnufied )

@dims, I can help with the storage parts, however, there are some steps related more to @kubernetes/cloud-provider-openstack-maintainers. As sig-storage we created https://docs.google.com/document/d/1j6uF9sOoGCQ9nNFVrdvCWyn77o9k3-758qsEj2K3BAY/edit# with the necessary steps. For OpenStack, I can't provide this:

• Making sure that the most common tools to install/upgrade Kubernetes on OpenStack install the CSI driver too. I don't even know what are the tools. Note that simple documentation "if you want Cinder CSI driver, enable XYZ" is not enough. The driver must be installed when a cluster is being upgraded to a version that has CSI migration enabled, the upgraded cluster gets broken otherwise.
• Add missing tests to CI. For OpenStack I miss topology tests, already found Fix translation of Cinder storage classess to CSI #98311.
• On the other hand, we don't need to support migration in Windows containers, because in-tree Cinder did not declare that it supports Windows.
• Some larger scale testing. CI can't cover everything. Since OpenStack is basically a Lego box that everyone assembles differently, report from multiple vendors would help too.

@dims, I can help with the storage parts, however, there are some steps related more to @kubernetes/cloud-provider-openstack-maintainers. As sig-storage we created https://docs.google.com/document/d/1j6uF9sOoGCQ9nNFVrdvCWyn77o9k3-758qsEj2K3BAY/edit# with the necessary steps. For OpenStack, I can't provide this:

• Making sure that the most common tools to install/upgrade Kubernetes on OpenStack install the CSI driver too. I don't even know what are the tools. Note that simple documentation "if you want Cinder CSI driver, enable XYZ" is not enough. The driver must be installed when a cluster is being upgraded to a version that has CSI migration enabled, the upgraded cluster gets broken otherwise.

This we will do by documenting in cluster-api-provider-openstack and updating docs.

• Add missing tests to CI. For OpenStack I miss topology tests, already found Fix translation of Cinder storage classess to CSI #98311.

There is a bunch of CI jobs in openlab, #98311 would have been caught there. So i am not worried.

• On the other hand, we don't need to support migration in Windows containers, because in-tree Cinder did not declare that it supports Windows.

Agree.

• Some larger scale testing. CI can't cover everything. Since OpenStack is basically a Lego box that everyone assembles differently, report from multiple vendors would help too.

We can't mandate and we can't wait. Other than RH, most of the other folks have been using extrernal cloud provider + CSI AFAIK. So we have to lead the way. If we have sufficient docs and make enough noise in openstack dev mailing list. that's enough.

thanks,
dims

This we will do by documenting in cluster-api-provider-openstack and updating docs.

As I wrote above, documentation may not be enough. Whatever tool a cluster admin uses to install or upgrade Kubernetes on OpenStack, this tool must install the CSI driver by default, otherwise the cluster is broken. Users don't read docs. I still haven't heard what's the most favorite tool to use on OpenStack. kops? kubeadm? Heat templates? Maybe they're all already fine, but someone must check it.

Add missing tests to CI. For OpenStack I miss topology tests, already found #98311.

There is a bunch of CI jobs in openlab, #98311 would have been caught there. So i am not worried.

No, openlab runs CSI migration tests and it did not catch this. I did not dig deeper, but the CI needs some fixes to enable topology. Does it even have multiple Cinder zones?

typo in the release note: my default -> by default

@dims: PR needs rebase.

No, openlab runs CSI migration tests and it did not catch this. I did not dig deeper, but the CI needs some fixes to enable topology. Does it even have multiple Cinder zones?

@jsafrane @dims we have checked this. It does run Topology tests , looks like default manifests of external-provisioner doesnt have Topology=true feature gate enabled, thats the reason it couldn't catch it . We are working on to fix it. Thanks.

And also, kops from release 1.19 , do install cinder driver by default , https://kops.sigs.k8s.io/releases/1.19-notes/#openstack-cinder-plugin

kops from release 1.19 , do install cinder driver by default

Thanks for confirmation! Is there any other widely used installation tool?

And also, kops from release 1.19 , do install cinder driver by default , https://kops.sigs.k8s.io/releases/1.19-notes/#openstack-cinder-plugin

It's not only about running topology tests and enabling topology in the provisioner. There are few tests than actually need to have a cluster with Cinder with multiple zones. Can openlab provide this? Since I ran into #98311, it seems to me that nobody has tested it before.

kops from release 1.19 , do install cinder driver by default

Thanks for confirmation! Is there any other widely used installation tool?

kubespray also do support openstack installation I suppose . But I dont have enough data to say which is widely used.

And also, kops from release 1.19 , do install cinder driver by default , https://kops.sigs.k8s.io/releases/1.19-notes/#openstack-cinder-plugin

It's not only about running topology tests and enabling topology in the provisioner. There are few tests than actually need to have a cluster with Cinder with multiple zones. Can openlab provide this? Since I ran into #98311, it seems to me that nobody has tested it before.

@jsafrane currently tests are not running with multi AZ. Though #98311 should be caught with signle AZ as well. we have already discussed and @chrigl is working with openlab to explore the possibility to enable for multi AZ and modify scripts accordingly.

For the record, I filed couple of issues that need to be fixed before we can enable this feature by default: https://github.com/kubernetes/cloud-provider-openstack/issues/created_by/jsafrane

(another view: https://github.com/orgs/kubernetes-csi/projects/40)

This is the first CSI driver that enables CSI migration. The plan is to have it in beta for two next releases to get feedback from users and have GA in 1.23 at the earliest.

/lgtm
/hold cancel

/milestone v1.21

/retest

@dims can you update the PR title to reflect that this is turning on the feature by default?

This PR should also note that - cloudprovider_openstack_api_request_duration_seconds and cloudprovider_openstack_api_request_errors will be renamed to openstack_api_request_duration_seconds and openstack_api_request_errors_total` - so as if there are any dashboards that use old metrics can be updated.