Sounds reasonable to me. Glad to have more Azure support.
On Mar 20, 2016 10:31 AM, "colrack" notifications@github.com wrote:

With this I want to track and share thoughts about introducing the ability
to provide support to attach a vhd as a volume on MS Azure.

With the recent kubernetes v1.2.0 the ability to use Azure File System as
a volume was introduced. I recently gave it a try on a cluster running on
CoreOS and with some workaround (see coreos/bugs#571
coreos/bugs#571 and coreos/bugs#358
coreos/bugs#358) it seems to work. However
Azure File Service has a lot of limitations (see
https://msdn.microsoft.com/en-us/library/azure/dn744326.aspx ) that
prevent to use it in many use cases.
Intro

In Azure we can attach a persistence disk to a vm. This is more or less
equal to what happens when you attach a Persistence Disk on GCE to vm or
what happens when you attach an EBS volume to a ec2 instance on AWS. When
you attach a vhd to a vm a new device is created in /dev and then you can
perform operations like format and mount; attaching and detaching a vhd
does not require to perform a reboot. Persistence disks in Azure are VHDs.
You can create them, attach and do other operations via REST API. VHDs are
saved as page blobs in an Azure Storage Account, inside a blob container
(see
https://azure.microsoft.com/en-us/documentation/articles/storage-introduction/
). In Azure you can attach a vhd to a single vm. So using a vhd as volume
would work like attaching to a pod a gcePersistentDisk or
awsElasticBlockStore and it would have more or less the same limitations,
including:

• the nodes on which pods are running must be Azure VMs
• those VMs need to be in the same location as the VHDs
• using a vhd on a pod controlled by a ReplicationController will fail
  unless the replica count is 0 or 1
• you can bind a volume backed by a vhd to a single pod, both in
  reading and writing

These limitations are due to the fact that, as reported before, you can
attach a vhd to a single vm. VHDs are mounted on a vm filesystem, and the
volume associated to this filesystem must be bound to a pod running on the
same vm. Hypothetically pods running in the same node could share the same
volume both in reading and writing, but this is a complicated and not so
useful scenario.
Work to do

Add in kubernetes/pkg/volume/azure_vhd/ the following:

• kubernetes/pkg/volume/azure_vhd/azure_file.go
• kubernetes/pkg/volume/azure_vhd/azure_file_test.go
• kubernetes/pkg/volume/azure_vhd/azure_util.go
• kubernetes/pkg/volume/azure_vhd/doc.go

You can see code in kubernetes/pkg/volume/, specially
kubernetes/pkg/volume/aws_ebs/ and kubernetes/pkg/volume/gce_pd/ since
the idea is the same.
Basically the code must call Azure api to attach the vhd to the node where
the pod is scheduled, mount the filesystem on that vm and share the volume
with the running container. Cloud provider logic is detached from volume
code, files are in kubernetes/pkg/cloudprovider/providers/, so stuff in
kubernetes/pkg/cloudprovider/providers/azure must be added and then used
from files listed above. The idea is that credentials to call Azure API
must be provided to masters in the cluster. This can be achieved by
creating a service principal (see
https://azure.microsoft.com/en-us/documentation/articles/resource-group-authenticate-service-principal/)
and let it access resources in the subscription where the kube cluster
resides. Azure SDK for go is here https://github.com/Azure/azure-sdk-f
or-go https://github.com/Azure/azure-sdk-for-go . The package to use is
arm/compute. Use func (VirtualMachinesClient) CreateOrUpdate
https://godoc.org/github.com/Azure/azure-sdk-for-go/arm/compute#VirtualMachinesClient.CreateOrUpdate
to update the vm passing appropriate params VirtualMachine ->
VirtualMachineProperties -> StorageProfile -> DataDisk -> VirtualHardDisk
.

An example pod would be like:

apiVersion: v1
kind: Pod
metadata:
name: test-vhd
spec:
containers:

• image: gcr.io/google_containers/test-webserver
  name: test-container
  volumeMounts:
  • mountPath: /test-vhd
    name: test-volume
    volumes:
• name: test-volume

  This Azure VHD must already exist.

  azureVhd:
  vhdUrl: https://myblobst.blob.core.windows.net/pdisks/c1.vhd
  fsType: ext4

Please comments and share your thoughts.
Feel free to point out if I'm missing something or if I'm wrong.

Ciao,
Carlo

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#23259

Hi @colrack,

I discussed this with some folks and it sounds like the disk attachment APIs for Azure are not very reliable and that it would be difficult to build a reliable feature on top of the current disk attachment APIs. I don't have a lot of specific, but I was told that another developer tried to build a feature on top of the disk attach APIs for Flocker and ran into significant resistance.

We have a feature in the works that would be more suited for these types of use-cases, but we don't have anything we can share about the feature or timeline currently.

Thanks @colemickens

Some time ago I went into this https://github.com/sedouard/azure-flocker-driver/ which at that time was a work in progress; I think this is the project you are talking about, discussions about issues here https://github.com/sedouard/azure-flocker-driver/issues/15
I don't have tried it so I can't confirm, but I suppose you at MS know more than me about this.
I'll try to stress azure disk-attach APIs to see this myself (seems like virtual machine will become unusable after repeated attaches and detaches of disk).

note this has to be sync'ed with #20262

Is it going to be in release 1.3 ?