@w17chm4n
There are no sig labels on this issue. Please add a sig label by:

1. mentioning a sig: @kubernetes/sig-<group-name>-<group-suffix>
   e.g., @kubernetes/contributor-experience-<group-suffix> to notify the contributor experience sig, OR

2. specifying the label manually: /sig <label>
   e.g., /sig scalability to apply the sig/scalability label

Note: Method 1 will trigger an email to the group. You can find the group list here and label list here.
The <group-suffix> in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals

@kubernetes/sig-node-bugs
/sig node

@w17chm4n: Reiterating the mentions to trigger a notification:
@kubernetes/sig-node-bugs.

One update - I can deploy this POD with no problems on 1.6.4-00 version of k8s.

@w17chm4n I am hitting the same bug on a freshly installed k8s cluster v1.7.0 running on Ubuntu 16.04.2 installed via kubeadm plus weave-net with a currently unsupported docker version.

root@94 ~ # docker version
Client:
 Version:      17.06.0-ce
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   02c1d87
 Built:        Fri Jun 23 21:23:31 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.06.0-ce
 API version:  1.30 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   02c1d87
 Built:        Fri Jun 23 21:19:04 2017
 OS/Arch:      linux/amd64
 Experimental: false

For me, downgrading to a supported docker version helped to resolve this issue (v1.12):

• https://docs.docker.com/cs-engine/1.12/#install-on-ubuntu-1404-lts-or-1604-lts

# https://kubernetes.io/docs/setup/independent/install-kubeadm/#installing-docker
On each of your machines, install Docker. Version 1.12 is recommended, but v1.10 and v1.11 are known to work as well. Versions 1.13 and 17.03+ have not yet been tested and verified by the Kubernetes node team

@bnerd - Thanks for the info!
@kubernetes/sig-node-bugs - Well we can confirm that kubernetes is not working properly with Docker 1.29 and 1.3 😛

@w17chm4n: Reiterating the mentions to trigger a notification:
@kubernetes/sig-node-bugs.

This is not really a bug since kubernetes doesn't support the docker version you use yet, but feel free to send a patch if you find the problem.

@yujuhong I was thinking about your comment and I think it is still a bug since downgrading Kubernetes to 1.6 fixes the problem while having the same setup as described above. So something have changed in 1.7. But yeah. Will try to investigate on my own.

I'm actually getting this same error with 1.7.1 and docker 1.13.1. Anyone have other thoughts on what the issue might be here? I found this so question (https://stackoverflow.com/questions/36545105/docker-couldnt-find-an-alternative-telinit-implementation-to-spawn) but it looks like everything is configured correctly.

$ docker version
Client:
 Version:      1.13.1
 API version:  1.26
 Go version:   go1.7.5
 Git commit:   092cba3
 Built:        Wed Feb  8 06:50:14 2017
 OS/Arch:      linux/amd64

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.1+coreos.0", GitCommit:"fdd5383472eb43e60d2222503f03c76445e49899", GitTreeState:"clean", BuildDate:"2017-07-18T19:44:47Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.1+coreos.0", GitCommit:"fdd5383472eb43e60d2222503f03c76445e49899", GitTreeState:"clean", BuildDate:"2017-07-18T19:44:47Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

$ cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.2 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.2 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

$ uname -a
Linux k5-master-0 4.4.0-78-generic #99-Ubuntu SMP Thu Apr 27 15:29:09 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

In kubernetes 1.7, under pid # 1, there is a "/ pause" process, and since init takes pid 1, it is no longer available. In earlier versions of kubernetes, pid # 1 was free .

/ # ps
PID USER TIME COMMAND
1 root 0:00 /pause
7 root 3:41 /tiller

Kubernetes now shares a single PID namespace among all containers in a pod when running with docker >= 1.13.1. This means processes can now signal processes in other containers in a pod, but it also means that the kubectl exec {pod} kill 1 pattern will cause the pod to be restarted rather than a single container. #45236

I can work around the problem using the --docker-disable-shared-pid option in the kubelet, however this not the desired solution of course.

@verb @yguo0905 just FYI, looks like some images have to be run as PID 1.

In 1.7 release, we enabled shared pid namespace if the docker version is docker 1.13 and beyond for debug pod feature. Do anyone know why those images, such as GlusterFS have to run as PID 1?

@verb we need to figure out how to resolve this conflict with shared pid namespace. If we cannot, we have to disable that setting for 1.8 since we scheduled to support docker 1.13 in 1.8 timeframe. I am currently assigning you this bug to come up a solution.

cc/ @abgworrall

gluster/gluster-centos is a container image that runs systemd so that it can run ntpd, crond, gssproxy, glusterd & sshd inside a single container. This isn't how Kubernetes is intended to be used and I don't know how much we should go out of our way to support it (this is a question for @dchen1107 & @yujuhong)

https://developers.redhat.com/blog/2016/09/13/running-systemd-in-a-non-privileged-container/ seems to be a good write up of the challenges of running systemd inside of docker. It also prevents Kubernetes from doing process management like:

• reporting state of processes
• signaling individual processes for graceful restart
• ingesting and display daemon output (to stdout/err)

Ideally we could change this many-processes-in-a-container to be many-containers-in-a-pod, something like this:

apiVersion: v1
kind: Pod
metadata:
  name: glusterpod
spec:
  restartPolicy: Never
  containers:
    - name: glusterd
      image: gluster/gluster-centos
      imagePullPolicy: Never
      command:
      - /usr/sbin/glusterd
      - -p
      - /var/run/glusterd.pid
      - --log-level
      - INFO
      - --no-daemon
      - --debug
      tty: true
      stdin: true
    - name: gssproxy
      image: gluster/gluster-centos
      imagePullPolicy: Never
      command:
      - /usr/sbin/gssproxy
      - --interactive
      - --debug

This doesn't need a privileged container and doesn't run sshd or ntpd, neither of which are needed in Kubernetes. This also doesn't run crond, but that may actually be needed by gluster. I've never used gluster. (I used --debug flags so I didn't have to investigate how to make these processes log to stdout)

If you must run systemd in a container, adding this to the container config will get systemd to run:

      env:
      - name: SYSTEMD_IGNORE_CHROOT
        value: "1"
      command:
      - /usr/lib/systemd/systemd
      - --system

This tells systemd to run in system mode even though it's not PID 1 and disables its chroot detection (which is unrelated to PID 1 but checked when invoked as "systemd", I guess).

@dchen1107 @verb afaict, running systemd init in a container is well recommended (https://developers.redhat.com/blog/2014/05/05/running-systemd-within-docker-container/ and https://developers.redhat.com/blog/2016/09/13/running-systemd-in-a-non-privileged-container/) at certain scenarios and a common practice. iic, there are enough containers who make use of init as its pid 1. If shared namespace which introduced in 1.7 breaks this, I believe its a backward compatibility break :(. I would invite @rhatdan here to share his thoughts as well. :)

@humblec Those articles are about how to work around shortcomings in vanilla docker that Kubernetes solves natively: resource sharing via pods, containing via SELinux, reaping orphaned zombies, etc.

My earlier question was along the lines of how much effort should go into supporting use cases that are not using Kubernetes as intended. (I'm actually asking, not trying to make a point.) Does Kubernetes commit to continuing support for every container image that's ever run on Kubernetes? For how long? Indefinitely? or can we change it over the course of multiple releases?

Anyway, this change is compatible with running an init system in a container. The problem is with images that assume they will only ever have pid == 1, which has so far been the case with docker due to a technical limitation, but it was never the case with other runtimes such as rkt, hyperv, etc.

Brainstorming other options:

1. Enable shared pid namespace and require users of systemd to set an environment variable in their Kubernetes config (or port the multi-process container to a multi-container pod).
2. Disable shared pid namespace and enable it in a future release where we have the same problem.
3. Design a solution where pid namespace is configurable per-pod, but not supported indefinitely. It would probably be implemented using annotations, and we will have to modify the Container Runtime Interface to support it. Over the course of multiple releases the default behavior would change but could be reverted based on the presence of an annotation. In a future release, users will have to change an annotation in their config to get the old behavior (rather than the environment variable). In a future future release, the annotation will stop working and users will have to set the environment variable.
4. Commit to supporting per-pod shared vs isolated pid namespaces indefinitely and adding it to the Kubernetes API. It doesn't make sense in some runtimes, so we probably can't require all runtimes to implement it, so it'll likely only see use with Docker. Last time we discussed it in SIG Node, @derekwaynecarr and @dchen1107 didn't want to support isolated pid namespaces indefinitely.

Yes running systemd in a container is a critical need for us. There are lots of situations where it is and will be used. Biggest use case if for people simply moving current workloads that run in a VM into a container.

1. If I have a service that runs fun under systemd in a VM. If I can simply take the init script and move it whole sale into a container, why wouldn't I want to do this. It's simple and does require a great deal of knowledge by the end user.

2. Multi service containers. Not every project can easily be moved into microservices. But they probably can run inside of containers. If we drop support for systemd running inside of containers, then we loose a big group of users.

3. Getting access to journald information inside of the container. Applications that write to syslog and directly to the journal. Currently in container world without systemd, this information is dropped on the floor. if I run systemd and journald, I can actually retrieve and use this information.

If the intension is to force all containers inside of a POD to run with the same PID Namespace? I think this is a mistake since it eliminates another really cool use case.

I would love to be able to run two containers inside of a pod, and INIT container with escallated privileges and a locked down container, with no privileges. I want to make sure the locked down container can not see the processes of the privileged container.

An example of this, might be a container that loads a kernel module that is needed by the locked down container. Another use case would be a buildah container. The INIT container mounts up a COW file system onto a directory on a location that the locked down container can write, The locked down container writes out the data for his image, when the container completes, the INIT container commits the data to the repo.

I believe allowing different containers inside of a POD to run with different security and isolated PIDs should not be removed.

@rhatdan none of the things you've mentioned are removed by this change, save using docker to hide other processes in the pod, and I'm not sure why that's a requirement. Is anyone currently using the "privileged sidecar" pattern you describe or is it hypothetical?

@dchen1107 @derekwaynecarr @yujuhong It just occurred to me that an interesting compromise here might be to only enable shared pid for multi-container pods. This would enable pod semantics for PID when it would make a difference while drop-in docker monoliths keep the PID 1 behavior they're expecting in docker. I'd have to double check, but I'm pretty sure that behavior could be implemented entirely in the dockershim with no API or CRI changes.

I think that is the best solution is to make the PID Namespace sharing optional.

@rhatdan - optional at what scope - cluster, node, pod. let's sync up before next sig-node and we can revisit if need be before 1.8 goes out. The main question right now is if pid namespace sharing is opt-in or opt-out. We need to weigh pro/cons of both.

I would expect it at the POD, since pods would have different requirements, not sure why you would want to have this at the cluster or node level.

@verb yes it works :)

@w17chm4n awesome, thanks!

I'd like to better understand the requirements of people objecting to this change (@rhatdan @jarrpa @humblec). I think there's a bit of confusion about what has changed, so let's start with some facts:

1. This container image runs on Kubernetes without changes to the image, but it requires (non-obvious) pod configuration.
2. There is nothing inherently incompatible between a shared pid namespace and a process manager like systemd. systemd and other init systems work with pid != 1, but may require special invocation.
3. Nothing in this issue is specific to Kubernetes. That is, this container image wouldn't work with vanilla docker PID namespace sharing, either.
4. This change does not affect securityContext

I can think of a few things that are objectionable about the change, but let's state them as requirements. For example, are these requirements for anyone?

• Kubernetes must run all containers published on docker hub with no additional pod configuration
• Kubernetes must run all systemd containers, regardless of how they are built, with zero additional pod configuration.
• Kubernetes must run this specific container image with zero additional configuration
• Kubernetes must officially support isolated PID namespaces between containers in a single pod across all runtimes (docker, rkt, hyperv, etc).
• Configuration changes are ok, but the --system flag to systemd cannot be used because the documentation says its not useful. I want to use systemd as intended.

Rather than having Kubernetes work around this container image, should we patch the Dockerfile to make it compatible with even the native docker shared-pid, unrelated to Kubernetes?

• Configuration changes are ok, but the --system flag to systemd cannot be used because the documentation says its not useful. I want to use systemd as intended.

We can ask the systemd guys why they recommend never using the --system flag, but I would guess one of the reasons is that processes showing up that were not created in the ancestry of systemd, could cause confusion. I would also guess that running more then one container with systemd --system would not be supported.

• Kubernetes must officially support isolated PID namespaces between containers in a single pod.

This is required so that you can run different containers with different security constraints in the same pod at the same time.

Your last comment makes the native docker shared-pid seem like it is the default, when actually it is the exception. almost no containers run with shared-pid normally. I guess in a side car container, it makes sense to support this scenario, but forcing this as the ONLY way to run containers is limiting.

This is required so that you can run different containers with different security constraints in the same pod at the same time.

This is the part I don't understand. Why? What does PID visibility have to do with security constraints? What am I missing?

almost no containers run with shared-pid normally

nor do they run with a shared network namespace, but we impose that on docker containers because that's the pod model.

An update from the last SIG Node: we'll disable this by default in 1.8 and revisit for 1.9

I'll use #41938 to track that.

@verb, if I have a privileged container process in the same pid namespace as a pid in a non priv container, then the container process in the non priv can attack the priv process. Having them in separate pid namespaces gives you better security.

Bottom line, you have a potential change where containers that expect to be run as PID 1, whether legitimately or not, could break when you make this change. Kubernetes in my opinion should make it easy to opt out of this behavior.

/priority critical-urgent

[MILESTONENOTIFIER] Milestone Labels Complete

@verb @w17chm4n

Issue label settings:

• sig/node: Issue will be escalated to these SIGs if needed.
• priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels.
• kind/bug: Fixes a bug discovered during the current release.

Update: #51634 is pending review and should be added to the v1.8 Milestone to resolve this issue.