docs: Self-hosted Kubelet proposal #23343

derekparker · 2016-03-22T17:49:34Z

Provides a proposal for changes needed with Kubernetes to allow for a
self-hosted Kubelet bootstrap.

k8s-bot · 2016-03-22T17:50:24Z

Can one of the admins verify that this patch is reasonable to test? (reply "ok to test", or if you trust the user, reply "add to whitelist")

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

derekparker · 2016-03-22T17:50:29Z

cc: @aaronlevy @philips @mikedanese @vishh @derekwaynecarr

k8s-bot · 2016-03-22T17:50:42Z

Can one of the admins verify that this patch is reasonable to test? (reply "ok to test", or if you trust the user, reply "add to whitelist")

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

k8s-bot · 2016-03-22T17:51:18Z

Can one of the admins verify that this patch is reasonable to test? (reply "ok to test", or if you trust the user, reply "add to whitelist")

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

mikedanese · 2016-03-22T17:53:13Z

@dchen1107

derekwaynecarr · 2016-03-22T21:04:54Z

docs/proposals/self-hosted-kubelet.md

+
+To expand on this, we envision a flow similar to the following:
+
+1. Systemd (or $init_system) continually runs “bootstrap” Kubelet in “runonce” mode with a file lock until it pulls down a “self-hosted” Kubelet pod and runs it.


assume --runonce-timeout results in a failure exit-code?

If there happens to be nothing scheduled to the node (hence hitting the --runonce-timeout) would we want that considered an error case?

In terms of the bootstrap kubelet / self-hosted kubelet pivot, I don't think coordinating around exit codes would be strictly necessary. Instead, the coordination point becomes "has another kubelet started", rather than "was I scheduled something". So $init could essentially loop on something like "if lockfile not acquired, start bootstrap kubelet; sleep X".

derekparker · 2016-04-04T16:28:46Z

cc: @vishh @mikedanese @dchen1107 @aaronlevy

To summarize the results of the in person meetings last Wednesday:

As per @vishh 's feedback we decided that from a system administrator perspective, have a service continually restart several times would likely be taken as an indication of failure, even though it only represents essentially a "loop iteration" until the bootstrap kubelet can pull down the "self-hosted" kubelet (again, this is until we get Taints and Tolerations).

So, instead of modifying the "runonce" code path to be able to contact an API Server, we will instead modify the default code path of the kubelet with a --bootstrap flag, which will indicate to the kubelet that after it acquires a file lock, it will wait for another kubelet to attempt to acquire that lock (via inotify) and then exit.

I will update the proposal to reflect this new approach. @vishh please let me know if I remembered anything about our discussion incorrectly :).

vishh · 2016-04-04T17:14:00Z

@derekparker: Thanks for the summary. LGTM.
The bootstrapping Kubelet will exit once it notices another process attempting to acquire the lock.
It will restart immediately though, and take over the node if the lock were to be released. There will be no restart loops.

During upgrades, the bootstrap kubelet might take over from the old version, before it let's the upgraded version run.

It is possible that the bootstrap kubelet version is incompatible with the newer versions that were run in the node. For example, the cgroup cofigurations might be incompatible.
In the beginning, we will require cluster admins to keep the configuration in sync. Since we want the bootstrap kubelet to come up and run even if the API server is not available, we should persist the configuration for bootstrap kubelet on the node.
Once we have checkpointing in kubelet, we will checkpoint the updated config and have the bootstrap kubelet use the updated config, if it were to take over.

aaronlevy · 2016-04-04T19:02:01Z

@vishh, coming back to this I'm still not fully clear on upgrade scenario.

bootstrap+upgrade scenario:

bootstrap-kubelet gets pod for v1-kubelet
v1-kubelet tries to acquire lock (inotify bootstrap kubelet)
bootstrap-kubelet dies due to inotify (but is restarted by init, and now waiting on lockfile)
v1-kubelet gets pod for upgraded v2-kubelet
- now v1-kubelet is running, v2-kubelet and bootstrap-kubelet waiting on file-lock
v1-kubelet pod definition is deleted, causing v1-kubelet to kill itself
If bootstrap-kubelet wins lock-file race and starts before v2-kubelet how does bootstrap-kubelet ever know to die (inotify events not queued)?

vishh · 2016-04-04T22:10:31Z

If bootstrap-kubelet wins lock-file race and starts before v2-kubelet how does bootstrap-kubelet ever know to die (inotify events not queued)?

Wouldn't the health checks on the kubelet pod end up restarting the kubelet?

aaronlevy · 2016-04-04T23:59:58Z

Ah right, that seems like it should work. Thanks.

derekparker · 2016-04-05T18:06:49Z

I have updated the proposal to reflect the latest discussions.

cc: @aaronlevy @dchen1107 @vishh @mikedanese @derekwaynecarr

vishh · 2016-04-12T17:25:26Z

docs/proposals/self-hosted-kubelet.md

+
+## Abstract
+
+In a self-hosted Kubernetes deployment, we have the initial bootstrap problem. This proposal presents a solution to the kubelet bootstrap, and assumes a functioning control plane, and a kubelet that can securely contact the API server.


What does a functioning control plane mean?

Could you describe the bootstrap problem?

When in the bootstrap process that a functioning control plan (assuming the apiserver) is needed?

In terms of the abstract, maybe we can elaborate a bit. Possibly something along the lines of:

When running self-hosted components, there needs to be a mechanism for pivoting from the initial bootstrap state to the kubernetes-managed (self-hosted) state. In the case of a self-hosted kubelet, this means pivoting from the initial kubelet defined & run on the host, to the kubelet pod which has been scheduled to the node.

This proposal presents a solution to the initial kubelet bootstrap, and the mechanism for pivoting to the self-hosted kubelet. This proposal assumes that the initial kubelet on the host is able to connect to a properly configured api-server.

Not sure if the above changes would answer the questions, but:

The bootstrap problem is essentially that we want the kubelet to be managed by kubernetes, but we need an initial kubelet to do that. So we need some mechanism for us to launch a kubelet, then give up control once a new kubelet has started.

A functioning apiserver would be required from the beginning (assuming no checkpointed pods I guess). Otherwise the initial kubelet would behave like any other kubelet without apiserver access.

I think it'd help explaining why we want kubelet to be managed by kubernetes.

It's unclear from this proposal that who's going to start the initial apiserver. I assume the bootstrap kubelet will have to start the apiserver pod based on the manifest files? In that case, the apiserver wasn't actually functioning at the time the kubelet was being started.

I guess I was thinking about this from the perspective of "in terms of this proposal, assume a functioning apiserver" as a means of keeping the discussion more focused. Ultimately the apiserver could be a static pod, or just a binary run directly on the host, or docker container outside k8s, etc.

But you're right that you wouldn't strictly need an apiserver to demonstrate the same functionality. The kubelet pod could be a static pod as well (would be an odd use-case, but should work assuming the kubelet pod doesn't need secrets/configMap/etc). So maybe just drop that line, as it is somewhat orthogonal?

Also, agree that it would be helpful to add a "motivations" section to cover reasons we want self-hosted kubelet.

vishh · 2016-04-12T20:57:48Z

@derekparker Completed review pass.

yujuhong · 2016-04-12T20:58:32Z

docs/proposals/self-hosted-kubelet.md

+# Proposal: Self-hosted kubelet
+
+## Abstract
+


nit: breaking up a paragraph with new lines will make commenting easier.

derekparker · 2016-04-18T16:42:24Z

Proposal has been updated.

cc: @aaronlevy @vishh @yujuhong @derekwaynecarr

derekparker · 2016-05-02T19:11:02Z

Updated proposal based on @vishh review.

@dchen1107 @derekwaynecarr any thoughts on this?

derekwaynecarr · 2016-05-09T17:20:55Z

@derekparker - apologies for the delay reviewing. I am ok with this as described.

philips · 2016-05-10T01:31:44Z

@vishh Can you ask @dchen1107 to review this? Or perhaps we can proceed without her review as you and @derekwaynecarr think that the approach is OK. We have been waiting for 12 days for a review.

k8s-bot · 2016-05-10T03:56:31Z

GCE e2e build/test passed for commit a7f4402.

k8s-github-robot · 2016-05-10T04:09:34Z

Automatic merge from submit-queue

bgrant0607 · 2016-05-12T20:31:40Z

@derekparker This PR should have been squashed before merge. Next time, please squash after LGTM.

derekparker · 2016-05-12T20:44:07Z

@bgrant0607 will do. FWIW I didn't really have much time / wasn't notified after lgtm tag was applied before the ok-to-merge tag was applied.

tmrts · 2016-05-12T20:46:20Z

adding ok-to-merge-after-squash label to the submit-queue might be useful

bgrant0607 · 2016-05-12T21:49:15Z

@tmrts File an issue on the contrib repo for that.

philips · 2016-05-14T10:20:56Z

PR for self-hosted kubelet --bootstrap flag is based on this proposal is here now: #25596

cheld · 2016-05-17T07:38:26Z

CC @batikanu, @zreigz

pwittrock · 2016-06-17T19:31:20Z

@aaronlevy
@vishh

Would you provide an update on the status for the documentation for this feature as well as add any PRs as they are created?

Not Started / In Progress / In Review / Done

Thanks
@pwittrock

aaronlevy · 2016-06-17T19:43:49Z

I do not believe any documentation has been started. Really the only work which has come out of this proposal thus far is adding the --exit-on-lock-contention flag to the kubelet.

I'm happy to add a blurb about the flag functionality as it stands if I can get a pointer to the best place to document flag functionality.

pwittrock · 2016-06-17T21:34:26Z

Is this a feature that would be used by anyone as is or that changes existing behavior? I see that the flag itself has been documented in the --help with Whether kubelet should exit upon lock-file contention.. If there are no other user facing changes, and this flag is not meant to be consumed by users yet, we can wait to document until the feature is complete.

pwittrock · 2016-06-17T22:27:52Z

From your response, it sounds like not doc changes are required for 1.3

aaronlevy · 2016-06-17T22:46:19Z

Probably not. We should probably update this proposal to reflect the changed flag name though (s/--bootstrap/--exit-on-lock-contention) - but assume that isn't a blocker.

…let-proposal Automatic merge from submit-queue docs: Self-hosted Kubelet proposal Provides a proposal for changes needed with Kubernetes to allow for a self-hosted Kubelet bootstrap.

googlebot added the cla: yes label Mar 22, 2016

k8s-github-robot assigned bgrant0607 Mar 22, 2016

k8s-github-robot added kind/design Categorizes issue or PR as related to design. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Mar 22, 2016

derekwaynecarr reviewed Mar 22, 2016
View reviewed changes

bgrant0607 assigned dchen1107 and unassigned bgrant0607 Mar 30, 2016

k8s-github-robot added the release-note-label-needed label Mar 31, 2016

vishh reviewed Apr 12, 2016
View reviewed changes

yujuhong reviewed Apr 12, 2016
View reviewed changes

docs: Update self-hosted proposal

a7f4402

derekparker force-pushed the self-hosted-kubelet-proposal branch from 5cb2e42 to a7f4402 Compare May 2, 2016 19:10

aaronlevy mentioned this pull request May 2, 2016

Use inotify-based self-hosted kubelet kubernetes-retired/bootkube#33

Closed

vishh added lgtm "Looks good to me", indicates that a PR is ready to be merged. release-note-none Denotes a PR that doesn't merit a release note. and removed release-note-label-needed labels May 10, 2016

k8s-github-robot added the needs-ok-to-merge label May 10, 2016

vishh added ok-to-merge labels May 10, 2016

k8s-github-robot removed the needs-ok-to-merge label May 10, 2016

k8s-github-robot merged commit 088694f into kubernetes:master May 10, 2016

derekparker mentioned this pull request May 13, 2016

kubelet: Optionally, have kubelet exit if lock file contention is observed, using --exit-on-lock-contention flag #25596

Merged

derekwaynecarr mentioned this pull request Oct 8, 2020

Moving lock related flags to Kubelet configuration #91877

Closed


		To expand on this, we envision a flow similar to the following:

		1. Systemd (or $init_system) continually runs “bootstrap” Kubelet in “runonce” mode with a file lock until it pulls down a “self-hosted” Kubelet pod and runs it.


		## Abstract

		In a self-hosted Kubernetes deployment, we have the initial bootstrap problem. This proposal presents a solution to the kubelet bootstrap, and assumes a functioning control plane, and a kubelet that can securely contact the API server.

docs: Self-hosted Kubelet proposal #23343

docs: Self-hosted Kubelet proposal #23343

Conversation

derekparker commented Mar 22, 2016

k8s-bot commented Mar 22, 2016

derekparker commented Mar 22, 2016

k8s-bot commented Mar 22, 2016

k8s-bot commented Mar 22, 2016

mikedanese commented Mar 22, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

derekparker commented Apr 4, 2016

vishh commented Apr 4, 2016

aaronlevy commented Apr 4, 2016

vishh commented Apr 4, 2016

aaronlevy commented Apr 4, 2016

derekparker commented Apr 5, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vishh commented Apr 12, 2016

Choose a reason for hiding this comment

derekparker commented Apr 18, 2016

derekparker commented May 2, 2016

derekwaynecarr commented May 9, 2016

philips commented May 10, 2016

k8s-bot commented May 10, 2016

k8s-github-robot commented May 10, 2016

bgrant0607 commented May 12, 2016

derekparker commented May 12, 2016

tmrts commented May 12, 2016 • edited Loading

bgrant0607 commented May 12, 2016

philips commented May 14, 2016

cheld commented May 17, 2016

pwittrock commented Jun 17, 2016

aaronlevy commented Jun 17, 2016

pwittrock commented Jun 17, 2016

pwittrock commented Jun 17, 2016

aaronlevy commented Jun 17, 2016

tmrts commented May 12, 2016 •

edited

Loading