Proposal for implementing init containers #23666

smarterclayton · 2016-03-31T02:27:59Z

Addresses #1589. Implemented in #23567. Docs in kubernetes/website#679

Init containers enable pod authors to perform tasks before their normal containers start. Each init container is started in order, and failing containers will prevent the application from starting.

smarterclayton · 2016-03-31T02:28:40Z

@kubernetes/sig-node @kubernetes/kube-api

timothysc · 2016-03-31T13:54:44Z

docs/proposals/container-init.md

+to start and that configuration is environment dependent, the image must be
+altered to add the necessary templating or retrieval.
+
+This proposal introduces the concept of an **init container**, one or more


This is a base case of DAG/workflow + pre&post-hook mechanics, except this proposal is missing the post bit.

@bgrant0607, @davidopp - Instead of speculating, I'll just ask ;-) Do you have a long term DAG/workflow model & hooks that you are planning?

If we want to do DAG, then we'd probably have data preconditions on
containers (the volume type container as proposed), and we'd need control
conditions. I may have missed that in Brian's comment about doing volume
type container (which form the edges) as the implication of the change.

If we implement the DAG, init containers are technically still useful and
easier to reason about total resources for. I believe in a future DAG
model you could convert init containers to a DAG defined solely in the
containers slice transparently, so one does not block the other.

On Thu, Mar 31, 2016 at 9:55 AM, Timothy St. Clair <notifications@github.com

wrote:

In docs/proposals/container-init.md
#23666 (comment)
:

+March 2016
+
+## Proposal and Motivation
+
+Within a pod there is a need to initialize local data or adapt to the current
+cluster environment that is not easily achieved in the current container model.
+Containers start in parallel after volumes are mounted, leaving no opportunity
+for coordination between containers without specialization of the image. If
+two containers need to share common initialization data, both images must
+be altered to cooperate using filesystem or network semantics, which introduces
+coupling between images. Likewise, if an image requires configuration in order
+to start and that configuration is environment dependent, the image must be
+altered to add the necessary templating or retrieval.
+
+This proposal introduces the concept of an init container, one or more

This is a base case of DAG/workflow + pre&post-hook mechanics, except this
proposal is missing the post bit.

@bgrant0607 https://github.com/bgrant0607, @davidopp
https://github.com/davidopp - Instead of speculating, I'll just ask ;-)
Do you have a long term DAG/workflow model & hooks that you are planning?

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
https://github.com/kubernetes/kubernetes/pull/23666/files/7efdc3d36197f1ca452339d49abbc87163fb0e75#r58056775

This really looks more like ~job/task/container-hooks despite the larger question. In other systems they are tied to life-cycle events:

fetch (which we may or may not care about)

pre/init (this proposal/setup)

update (which we don't really do, but useful in batching/restart)

evict (sometimes treated separately, billing and what not)

exit (we probably want/tear-down)

dunno if you want to address here or explicitly defer?

I am strongly opposed to DAG.

Other proposed hooks include preStart and postStop, at the container level: #140

Is "PreStart" container hooks the same as a pre-start pod hook?

A pre start container hook would just be some bash that executes a binary in the container right? If I need to insert the binary in my container, I might as well invoke it in an entrypoint script. How much value does it add?

Instead of stacking init containers at the pod level, we could just attach them to the specific container they're initializaing and call them PreStart container hooks. This feels less useful in a pod context, I'd want to wait till all containers in my pod are initialized before starting any part of the app.

@bprashanth responding to vish i was addressing renaming. You seem to be addressing functional gap? I thought we seemed to be ok on init container as pre app-container start, merely naming.

Ack. Pardon my confusion, I was just making sure we weren't doing a pre start container hook instead.

@smarterclayton So what is the final decision? Is it going to stay as init containers or be renamed to pre-start containers?

I'm open - was waiting to see if others had a preference.

derekwaynecarr · 2016-03-31T17:36:52Z

Ensuring I do not forget to come back and read this in more detail when not debugging node issues. (cc @derekwaynecarr)

yifan-gu · 2016-03-31T18:44:50Z

@jonboulle @philips We need to think about how this can be implemented in rkt without introducing container level intefaces to rkt for now. e.g. Introducing dependencies between the apps and let systemd of the pod to handle it?

cc @dchen1107 @vishh @sjpotter

smarterclayton · 2016-03-31T19:25:02Z

I'll incorporate whatever plan you guys have - I had assumed systemd
dependencies would be a good angle but wanted to let you guys weigh in
before going any further.

On Thu, Mar 31, 2016 at 2:45 PM, Yifan Gu notifications@github.com wrote:

@jonboulle https://github.com/jonboulle @philips
https://github.com/philips We need to think about how this can be
implemented in rkt without introducing container level intefaces to rkt for
now. e.g. Introducing dependencies between the apps and let systemd of the
pod to handle it?

cc @dchen1107 https://github.com/dchen1107 @vishh
https://github.com/vishh @sjpotter https://github.com/sjpotter

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#23666 (comment)

timstclair · 2016-03-31T21:17:26Z

docs/proposals/container-init.md

+  block startup of containers
+* Running a "pre-pod" would defeat the purpose of the pod being an atomic
+  unit of scheduling.
+


What about adding a StartCondition? The start condition could be things like, Container A is ready, Container B is successful, or Container C is exited. Pros: (maybe) fewer new concepts; can represent arbitrary dags. Cons: adds lifecycle complexity, specifically complicates the restart policy; startup order less predictable.
Is this what @timothysc proposed above?

Example issue: We support image updates. What should happen if we update the image of a container that other containers depend on? For that matter, if a container fails and restarts, what should happen to dependent containers?

bgrant0607 · 2016-04-01T06:23:21Z

docs/proposals/container-init.md

+## Design Points
+
+* Init containers should be able to:
+  * Perform initialization of shared volumes


I still think #831 is preferable for that.

For shared volumes do are you concerned that this proposal would block, limit, or complicate the efficacy / use of shared volumes? I would view the container volume as additive on top of this, but with some orthogonal concerns this proposal does not require changes to address (long running "init" container, implicit start order, precondition order, may require chain restarts on docker if the container provider is restarted, much more complex changes to the kubelet and volume code). Container volumes (vs image volumes) require a virtual DAG implementation for restart, except that users cannot set preconditions. Would container volumes that run forever (fuse providing volume) be considered an init container? I'd argue it does not sound like that, so if it was a regular container we would have a limited DAG in the pod containers spec. If we have a limited DAG, users can then construct deeper dependencies and abuse the DAG.

I'll amend that to we would have a start / readiness precondition DAG if a fuse container was modeled with container volume defining the dependency.

If init containers execute after all volumes have been mounted and initialized, it shouldn't preclude implementation of #831, no.

yifan-gu · 2016-04-06T06:19:12Z

Other proposed hooks include preStart and postStop, at the container level: #140

@bgrant0607 @smarterclayton I also feel this is something can be implemented by preStart, even you have multiple containers that depends on the init container.

In this case the preStart should run in the container's namespace, which is something like what's defined in appc's pre-start.

One concern might be the app container's image doesn't have the binaries available for the hooks, but I don't really bother about this, as if users wants to run the hooks, they need to have the binary available anyway, either in the app container or in the init container. To me it seems have the binaries available in the app's image reduces the overhead then using a particular image for the init container.

So I am curious what values init containers add besides preStart hooks? Of course, one obvious reason could be that docker doesn't have preStart hooks today...

cc @philips @jonboulle

smarterclayton · 2016-04-06T16:09:20Z

A big design point for init containers is to not have to have the binaries
in the app image. It's a composition pattern. Also, init containers may
need to run under different security privileges or have access to different
secrets than regular containers.

On Wed, Apr 6, 2016 at 2:19 AM, Yifan Gu notifications@github.com wrote:

Other proposed hooks include preStart and postStop, at the container
level: #140 #140

@bgrant0607 https://github.com/bgrant0607 @smarterclayton
https://github.com/smarterclayton I also feel this is something can be
implemented by preStart, even you have multiple containers that depends on
the init container.

In this case the preStart should run in the container's namespace, which
is something like what's defined in appc's pre-start
https://github.com/appc/spec/blob/master/spec/aci.md.

One concern might be the app container's image doesn't have the binaries
available for the hooks, but I don't really bother about this, as if users
wants to run the hooks, they need to have the binary available anyway,
either in the app container or in the init container. To me it seems have
the binaries available in the app's image reduces the overhead then using a
particular image for the init container.

So I am curious what values init containers add besides preStart hooks? Of
course, one obvious reason could be that docker doesn't have preStart hooks
today...

cc @philips https://github.com/philips @jonboulle
https://github.com/jonboulle

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#23666 (comment)

bgrant0607 · 2016-04-08T20:26:18Z

docs/proposals/container-init.md

+  * Download binaries that will be used in app containers as execution targets
+  * Inject configuration or extension capability to generic images at startup
+  * Perform complex templating of information avaliable in the local environment
+  * Register the pod with other components of the system


I think there are (at least) 3 categories of "init" actions:

Fetch data into volumes. Better served by Image volumes and container volumes #831 IMO, but if this is the only hammer we're building, it could also pound that screw.

Perform context-specific transformations of volume data, likely out-of-place, and other one-time config generation (perhaps using downward-API data). I thik this is usually container-specific rather than volume-specific, and perhaps could be done with a preStart hook (maybe leveraging a script mounted from a volume), if we had preStart hooks. Again, could be done by an init container in lieu of preStart hooks, though the separation of the init container and consuming container in the spec could be awkward.

Perform "read-only" pre-start actions, like registration and delaying startup. Ideal for a pod-level init container.

dchen1107 · 2016-06-03T18:52:14Z

What are the open issues left here for closing this proposal? I raised QoS issues above, and thought we reached an agreement already and the proposal from @vishh already covered that based our discussion and merged. Also QoS issue shouldn't be blocker here.

smarterclayton · 2016-06-03T18:59:45Z

I don't think there is anything left except to document the current
discussion in the PR.

On Jun 3, 2016, at 2:52 PM, Dawn Chen notifications@github.com wrote:

What are the open issues left here for closing this proposal? I raised QoS
issues above, and thought we reached an agreement already and the proposal
from @vishh https://github.com/vishh already covered that based our
discussion and merged. Also QoS issue shouldn't be blocker here.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#23666 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/ABG_pycH-5R_5Bu1ULo_7MHpDKci3OFSks5qIHfvgaJpZM4H8XbD
.

0xmichalis · 2016-06-05T16:00:38Z

docs/proposals/container-init.md

+  * Using a volume container that does not populate a volume to delay pod start
+    (in the absence of init containers) would be an abuse of the goal of volume
+    containers.
+* Container pre-start hooks are not sufficient for all initialization cases:


Since pre-start hooks are not implemented yet, link to the issue: #140

smarterclayton · 2016-06-05T23:32:16Z

Updated with all feedback to date (noted that pod QoS is higher of init container QoS or regular container QoS, which was part of the discussion above).

pwittrock · 2016-06-17T19:29:12Z

@smarterclayton
@bprashanth
@dchen1107

Would you provide an update on the status for the documentation for this feature as well as add any PRs as they are created?

Not Started / In Progress / In Review / Done

Docs Instructions for v1.3

Thanks
@pwittrock

bprashanth · 2016-06-17T19:32:00Z

I totally forgot about this proposal since we got the implementation in.Giving it a final read through, I think it's good to go in though.

pwittrock · 2016-06-17T20:42:37Z

Docs status?

smarterclayton · 2016-06-17T21:37:24Z

Docs are waiting for review in kubernetes/website#679

smarterclayton · 2016-06-17T21:54:11Z

Added release note.

k8s-bot · 2016-06-17T23:54:38Z

GCE e2e build/test passed for commit bdde25c.

k8s-bot · 2016-06-18T00:02:03Z

GCE e2e build/test passed for commit bdde25c.

k8s-github-robot · 2016-06-18T00:02:16Z

Automatic merge from submit-queue

mqliang · 2016-10-08T06:41:07Z

@smarterclayton @bgrant0607

Why not add general Pod level life cycle hook? AFAIK, we have container level life cycle hook, Pod level life cycle hook is also very helpful.

IMO, InitContainers can implement as a kind of Pod level PreStart hook. And we have some case need Pod level PreStop hook support, it's ugly to add a PostContainers. A general Pod level life cycle hook sounds better.

smarterclayton · 2016-10-08T14:20:32Z

Hooks run in containers. For there to be a post stop hook, there must be a
post stop container.

Can you describe a usecase for a post-stop pod hook so we can discuss
something more concrete?

On Oct 8, 2016, at 12:41 AM, Liang Mingqiang notifications@github.com
wrote:

@smarterclayton https://github.com/smarterclayton @bgrant0607
https://github.com/bgrant0607

Why not add general Pod level life cycle hook? AFAIK, we have container
level life cycle hook, Pod level life cycle hook is also very helpful.

IMO, InitContainer can implement as a kind of Pod level PreStart hook. And
we have some case need Pod level PreStop hook support, it's ugly to add a
PostContainer. A general Pod level life cycle hook sounds better.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#23666 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABG_p5cABtQDwoI_0K1NNOsU7WweKwpFks5qxzsQgaJpZM4H8XbD
.

mqliang · 2016-10-08T14:39:23Z

@smarterclayton

Can you describe a usecase for a post-stop pod hook so we can discuss
something more concrete?

For example, I have a Pod P, which running a container C. I want to commit container C and push it into registry when Pod exit. Then, I need a Pod level pre/post stop hook. In the pre/post stop hook, I run a new container which could access docker daemon(by mount /var/run/docker.sock into container) to commit and push container C in registry

smarterclayton · 2016-10-08T23:21:09Z

I'd recommend you do this via a privileged container that observes
termination of the "container to commit" then either commits or aborts.
That's our current working plan for openshift containers.

After all, you need to know success or failure.

On Oct 8, 2016, at 8:39 AM, Liang Mingqiang notifications@github.com
wrote:

@smarterclayton https://github.com/smarterclayton

Can you describe a usecase for a post-stop pod hook so we can discuss
something more concrete?

For example, I have a Pod P, which running a container C. I want to commit
container C and push it into registry when Pod exit. Then, I need a Pod
level pre/post stop hook. In the pre/post stop hook, I run a new container
which could access docker daemon(by mount /var/run/docker.sock into
container) to commit and push container C in registry

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#23666 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABG_p_1bcv0bIwNo7m9fyrfvz7NpsIOWks5qx6sogaJpZM4H8XbD
.

googlebot added the cla: yes label Mar 31, 2016

smarterclayton added the kind/design Categorizes issue or PR as related to design. label Mar 31, 2016

smarterclayton added the area/kubelet label Mar 31, 2016

k8s-github-robot assigned bgrant0607 Mar 31, 2016

k8s-github-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Mar 31, 2016

smarterclayton mentioned this pull request Mar 31, 2016

Add init containers to pods #23567

Merged

timothysc reviewed Mar 31, 2016
View reviewed changes

timstclair reviewed Mar 31, 2016
View reviewed changes

k8s-github-robot added the release-note-label-needed label Mar 31, 2016

bgrant0607 reviewed Apr 1, 2016
View reviewed changes

bgrant0607 reviewed Apr 8, 2016
View reviewed changes

0xmichalis reviewed Jun 5, 2016
View reviewed changes

smarterclayton force-pushed the initctrproposal branch 2 times, most recently from 0ce2d72 to f46a2f7 Compare June 5, 2016 23:31

lukaszo mentioned this pull request Jun 9, 2016

Fix kubectl logs for init containers #27042

Merged

bprashanth added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 17, 2016

k8s-github-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 17, 2016

smarterclayton force-pushed the initctrproposal branch from f46a2f7 to bdde25c Compare June 17, 2016 21:38

Proposal for implementing init containers

bdde25c

smarterclayton added release-note-experimental lgtm "Looks good to me", indicates that a PR is ready to be merged. and removed release-note-label-needed labels Jun 17, 2016

bprashanth added this to the v1.3 milestone Jun 17, 2016

k8s-github-robot merged commit d03a5fa into kubernetes:master Jun 18, 2016

Proposal for implementing init containers #23666

Proposal for implementing init containers #23666

Conversation

smarterclayton commented Mar 31, 2016 • edited Loading

smarterclayton commented Mar 31, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

smarterclayton Apr 22, 2016 via email

Choose a reason for hiding this comment

derekwaynecarr commented Mar 31, 2016

yifan-gu commented Mar 31, 2016

smarterclayton commented Mar 31, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

smarterclayton Apr 1, 2016 via email

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yifan-gu commented Apr 6, 2016

smarterclayton commented Apr 6, 2016

Choose a reason for hiding this comment

dchen1107 commented Jun 3, 2016

smarterclayton commented Jun 3, 2016

Choose a reason for hiding this comment

smarterclayton commented Jun 5, 2016

pwittrock commented Jun 17, 2016 • edited Loading

bprashanth commented Jun 17, 2016

pwittrock commented Jun 17, 2016

smarterclayton commented Jun 17, 2016

smarterclayton commented Jun 17, 2016

k8s-bot commented Jun 17, 2016

k8s-bot commented Jun 18, 2016

k8s-github-robot commented Jun 18, 2016

mqliang commented Oct 8, 2016 • edited Loading

smarterclayton commented Oct 8, 2016

mqliang commented Oct 8, 2016

smarterclayton commented Oct 8, 2016

smarterclayton commented Mar 31, 2016 •

edited

Loading

pwittrock commented Jun 17, 2016 •

edited

Loading

mqliang commented Oct 8, 2016 •

edited

Loading