Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Capability to specify mount propagation mode of per volume with docker #20698

Closed
wants to merge 1 commit into from
Closed

Conversation

m3ngyang
Copy link
Contributor

@m3ngyang m3ngyang commented Feb 5, 2016

Volume mount propagation mode has been supported since docker 1.10.0, but this function is absent in current kubelet.
This patch set allows one to specify mount propagation setting of a volume. Allowed values are "shared" or "slave" or "private", via introducing a new filed called "Propagation" in type VolumeMount in api/types


This change is Reviewable

@k8s-bot
Copy link

k8s-bot commented Feb 5, 2016

Can one of the admins verify that this patch is reasonable to test? (reply "ok to test", or if you trust the user, reply "add to whitelist")

If this message is too spammy, please complain to ixdy.

@m3ngyang
Copy link
Contributor Author

m3ngyang commented Feb 5, 2016

@k8s-bot ok to test

@k8s-bot
Copy link

k8s-bot commented Feb 5, 2016

Can one of the admins verify that this patch is reasonable to test? (reply "ok to test", or if you trust the user, reply "add to whitelist")

If this message is too spammy, please complain to ixdy.

@k8s-github-robot k8s-github-robot added kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Feb 5, 2016
@thockin
Copy link
Member

thockin commented Feb 5, 2016

What is a user going to do with this flag? Mount prop. flags are INCREDIBLY complicated - what behavior do people need that this satisfies?

@m3ngyang
Copy link
Contributor Author

m3ngyang commented Feb 5, 2016

Our team attempts to containerize flocker with zfs as back-end storage, and containers in the same flocker node need to read/write and share the same mounted volume. Without this flag, the volume mount propagation mode cannot be specified between the host and the container, and then the volume mount of each container would be isolated by namespace from each other.
By default, the flag would be set with empty string and parsed as 'rprivate' in docker.
If someone chooses to use the propagation flag, he should be familiar with shard mount and responsible for the result himself.@thockin

@k8s-github-robot
Copy link

PR needs rebase

@k8s-github-robot k8s-github-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 5, 2016
@thockin
Copy link
Member

thockin commented Feb 5, 2016

@pmorie @swagiaal or whomever is looking at containerized kubelet now that mount.prop is available in Docker. Is the plan to enable shared by default? I don't like this as a facet of our API if we can avoid it.

@m3ngyang
Copy link
Contributor Author

m3ngyang commented Feb 5, 2016

@thockin besides shared mode, Docker 1.10.0 also supports slave and private mode.
fyi moby/moby#17034

@thockin
Copy link
Member

thockin commented Feb 5, 2016

I know what it supports, we've been watching it for a long time :)

My point is that this is a REALLY complicated API feature that 0.001% of
users will ever need, and I'd rather DTRT by default than offer a flag.

On Fri, Feb 5, 2016 at 8:34 AM, Kaffa notifications@github.com wrote:

@thockin https://github.com/thockin besides shared mode, Docker 1.10.0
also support slave and private mode.
fyi moby/moby#17034 http://url


Reply to this email directly or view it on GitHub
#20698 (comment)
.

@pmorie
Copy link
Member

pmorie commented Feb 5, 2016

@Kaffa-MY I agree with @thockin that we need to be careful about exposing an API surface for this. Currently the default is private in raw docker.

@Kaffa-MY I'm curious about what your use-case for this is.

@ghost
Copy link

ghost commented Feb 5, 2016

Hmm.. I wonder what would be a good way to expose this if not through the API.

Atomic Host has the same use case which is basically containerizing the mount tools so they don't have to be installed on the minimal host.

Is there a way to make this available to volume plugins without putting it in the API ? That way the change can be made to the flocker plugin as I don't think there is a general use case of pod's needing to propagate mounts.

@dchen1107
Copy link
Member

cc/ me

@pmorie
Copy link
Member

pmorie commented Feb 5, 2016

So here's one use-case I can imagine: say that we want to containerize mount operations and we want to run those containers in pods.

@thockin
Copy link
Member

thockin commented Feb 5, 2016

Isn't that what Sami said?

I thought the plan was to move all volumes to shared by default
specifically to allow mount helpers, FUSE daemons, and containerized kubelet

On Fri, Feb 5, 2016 at 3:09 PM, Paul Morie notifications@github.com wrote:

So here's one use-case I can imagine: say that we want to containerize
mount operations and we want to run those containers in pods.


Reply to this email directly or view it on GitHub
#20698 (comment)
.

@pmorie
Copy link
Member

pmorie commented Feb 5, 2016

@kubernetes/rh-storage @kubernetes/rh-cluster-infra

@pmorie
Copy link
Member

pmorie commented Feb 6, 2016

It is, @thockin, I hadn't seen sami's comment load in the browser window I had this in.

@m3ngyang
Copy link
Contributor Author

m3ngyang commented Feb 6, 2016

@pmorie I'm containerizing flocker with zfs as backend storage, containers in the same pod, which constitute a whole flocker agent, need to share the same volume and the mount information.
@thockin If the shared were set as the default propagation mode of volume mount and the API were not exposed, how to deal with the situation where isolation is needed?

@k8s-github-robot k8s-github-robot added kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API kind/design Categorizes issue or PR as related to design. kind/new-api size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Feb 6, 2016
@pmorie
Copy link
Member

pmorie commented Feb 6, 2016

@Kaffa-MY @thockin

The default propagation mode in docker is private.

@k8s-github-robot
Copy link

PR needs rebase

@k8s-bot
Copy link

k8s-bot commented Aug 2, 2016

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

@luxas luxas added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed do-not-merge DEPRECATED. Indicates that a PR should not merge. Label can only be manually applied/removed. release-note-label-needed labels Aug 9, 2016
@luxas
Copy link
Member

luxas commented Aug 9, 2016

This would be VERY valuable to have in v1.4, given all the @kubernetes/sig-cluster-lifecycle work.
It would make it possible to run kubelet as a pod easily, where /var/lib/kubelet has to be a shared mount.

@luxas
Copy link
Member

luxas commented Aug 9, 2016

@thockin @pmorie @rootfs Any chance you'd pick it up?
It should be documented it only works with docker; but on the other hand, if we some day want to support rkt fly; it won't be supported by docker.

@andrewmchen
Copy link

+1. Would also really appreciate this feature.

@andrewmchen
Copy link

@Kaffa-MY Is the full PR including the implementation public?

@k8s-bot
Copy link

k8s-bot commented Aug 11, 2016

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

@vishh
Copy link
Contributor

vishh commented Aug 11, 2016

+1 for @thockin's comments. Why can't hostpath volumes be mounted as shared by default? Assuming that not all containers will have the privilege to mount filesystems, are there drawbacks to mounting shared by default?

@luxas Regarding "It would make it possible to run kubelet as a pod easily"

Why is kubelet being run as a pod? I don't think this is an operating mode that @kubernetes/sig-node is willing to support.

@luxas
Copy link
Member

luxas commented Aug 11, 2016

@vishh Check out self-hosting. It makes a lot of sense.
And running kubelet in a container with a shared mount works really well.
Kubelet in container w/ shared mount != --containerized, which have had problems.

See kubernetes-retired/kube-deploy#189 for ref.
Most of the fails there was due to the version skew (v1.3.4 vs e2e.test v1.4.0 alpha.2)

I guess the main drawback with default shared mounts is that it will be a docker-specific behaviour then. Otherwise, we could leave it as a docker-specific option that user that understand the implications (and the need for --privileged for it to work) could turn on. Perfect for storafe drivers for example.

We might even be able to containerize the storage driver itself, and then kubelet just could pull and use a such Pod for mounting volumes (like we're using overlay network plugins in a daemonset today). cc @pmorie

@kfox1111
Copy link

Yeah. I plan on using it for making a daemonset with cvmfs in it instead of flexvolumes.

@k8s-bot
Copy link

k8s-bot commented Aug 16, 2016

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

@k8s-bot
Copy link

k8s-bot commented Aug 21, 2016

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

@porridge
Copy link
Member

+1 to #20698 (comment) - this would make it possible to use Cloud SQL proxy's -fuse feature.

@m3ngyang
Copy link
Contributor Author

@andrewmchen not including implementation, this is just an api proposal

PropagationShared PropagationMode = "shared"
PropagationRShared PropagationMode = "rshared"
)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pmorie is this emun ok?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think based on the discussion here, the next step is to figure out the answer to the following two questions:

  1. Can this be solved by a single sane default for volume mounts?
  2. If not 1, what's the minimal feature we need to expose to users to enable people the most, and what's a thinner API for it.

It's possible that this is the correct minimal thing to expose, but based on the discussion here, it needs to justify itself (and I personally think it is not).

Based on @thockin's comment and concerns (which I share), I think the next step here is to write a proposal explaining what the options are, what the use cases are, and explaining exactly why a given option is the right one.
I think iterating on a .md proposal will be more productive than iterating on the exact code / api-changes.

@k8s-bot
Copy link

k8s-bot commented Aug 30, 2016

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message will repeat several times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

@k8s-github-robot
Copy link

This PR hasn't been active in 30 days. It will be closed in 59 days (Nov 27, 2016).

cc @Kaffa-MY @thockin

You can add 'keep-open' label to prevent this from happening, or add a comment to keep it open another 90 days

@thockin
Copy link
Member

thockin commented Oct 1, 2016

Either this or #31504 has to be closed. Given that #31504 is more limited in scope, I vote to close this one (no offense). Scream if I got it wrong (typical :)

@thockin thockin closed this Oct 1, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API release-note Denotes a PR that will be considered when it comes time to generate release notes. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.