Implement the `kubeadm upgrade` command #48899

luxas · 2017-07-13T20:06:16Z

What this PR does / why we need it:

Implements the kubeadm upgrades proposal: https://docs.google.com/document/d/1PRrC2tvB-p7sotIA5rnHy5WAOGdJJOIXPPv23hUFGrY/edit#

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #

fixes: kubernetes/kubeadm#14

Special notes for your reviewer:

I'm gonna split out changes not directly related to the upgrade procedure into separate PRs as dependencies as we go.
Now ready for review. Please look at cmd/kubeadm/app/phases/upgrade first and give feedback on that. The rest kind of follows.

Release note:

Implemented `kubeadm upgrade plan` for checking whether you can upgrade your cluster to a newer version
Implemented `kubeadm upgrade apply` for upgrading your cluster from one version to an other

cc @fabriziopandini @kubernetes/sig-cluster-lifecycle-pr-reviews @craigtracey @mattmoyer

timothysc

Needs more input on the UX side before we go too far.

timothysc · 2017-07-17T15:16:46Z

cmd/kubeadm/app/cmd/upgrade/apply.go

+
+func NewCmdApply() *cobra.Command {
+	cmd := &cobra.Command{
+		Use:   "apply",


So I'm not sold on apply vs. :

> kubectl upgrade or > kubectl upgrade --dry-run

timothysc · 2017-07-18T19:04:16Z

Per design doc discussion there will be an apply and plan and no dry-run.

https://docs.google.com/document/d/1PRrC2tvB-p7sotIA5rnHy5WAOGdJJOIXPPv23hUFGrY/edit?disco=AAAAA_en9VI&ts=596e54c5

/cc @jbeda

luxas · 2017-07-26T20:54:03Z

ping @timothysc @andrewrynhard @kubernetes/sig-cluster-lifecycle-feature-requests

PTAL if you want/like. This is not the end state, it's work in progress.
There's a lot of functionality obviously, so I'm gonna split this up in separate PRs later on.
But first I'm gonna do everything in obe branch here, that people can easily checkout and test.

When I've written all necessary unit tests, added all necessary functionality for a MVP and cleaned up, this is ready for the real review.
However, it'd be great if you want to take a quick pass on a high level so I can catch things sooner rather than later.

So feel free to compile yourself and test :)
Only thing that's yet missing is the actual rolling upgrade code for the control plane. Not sure if I should do it the hacky way now (I probably will, just to get things off the ground) or wait until we have "add first, then delete" update strategy for DaemonSets.

WDYT?

luxas · 2017-08-04T15:03:46Z

This is now ready for review 🎉! Both plan and apply works, and most of the functionality is there.
I'll break out dependencies on refactoring and such things later into separate PRs.
Please review cmd/kubeadm/app/phases/upgrade and the UX first

The UX should/will be greatly improved yet. Showing a snapshot of current state here:

UX:

$ # --apiserver-bind-port is there to show that configuration is preserved with the upgrade. If it wouldn't, things would break in between when the API server would start listening on the default 6443, but this works.
$ # I've put kubeadm built from this PR in ./kubeadm
$ ./kubeadm init --kubernetes-version v1.7.1 --apiserver-bind-port 443
$ kubectl apply -f https://git.io/weave-kube-1.6
$ ./kubeadm alpha upgrade plan
[upgrade] Making sure the cluster is healthy:
[upgrade/health] Checking API Server health: Healthy
[upgrade/health] Checking Node health: All Nodes are healthy
[upgrade/health] Checking if control plane is Static Pod-hosted or Self-Hosted: Static Pod-hosted.
[upgrade/health] NOTE: kubeadm will upgrade your Static Pod-hosted control plane to a Self-Hosted one when upgrading if --feature-gates=SelfHosting=true is set (which is the default)
[upgrade/health] If you strictly want to continue using a Static Pod-hosted control plane, set --feature-gates=SelfHosting=true when running 'kubeadm upgrade apply'
[upgrade/health] Checking Static Pod manifests exists on disk: All required Static Pod manifests exist on disk
[upgrade] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster (you can get this with 'kubectl -n kube-system get cm kubeadm-config -oyaml')
[upgrade] Fetching available versions to upgrade to:
[upgrade/versions] Cluster version: v1.7.1
[upgrade/versions] kubeadm version: v1.8.0-alpha.2.789+11f48dc291fe93
[upgrade/versions] Latest stable version: v1.7.3
[upgrade/versions] Latest version in the v1.7 series: v1.7.3

Components that must be upgraded manually after you've upgraded the control plane with `kubeadm upgrade apply`:
COMPONENT   CURRENT      AVAILABLE
Kubelet     1 x v1.7.0   v1.7.3

Upgrade to the latest version in the v1.7 series:

COMPONENT            CURRENT   AVAILABLE
API Server           v1.7.1    v1.7.3
Controller Manager   v1.7.1    v1.7.3
Scheduler            v1.7.1    v1.7.3
Kube Proxy           v1.7.1    v1.7.3
Kube DNS             1.14.4    1.14.4

You can now apply the upgrade by executing the following command:

	kubeadm upgrade apply --version v1.7.3

_____________________________________________________________________

$ ./kubeadm alpha upgrade apply --version v1.7.3
[upgrade] Making sure the cluster is healthy:
[upgrade/health] Checking API Server health: Healthy
[upgrade/health] Checking Node health: All Nodes are healthy
[upgrade/health] Checking if control plane is Static Pod-hosted or Self-Hosted: Static Pod-hosted.
[upgrade/health] NOTE: kubeadm will upgrade your Static Pod-hosted control plane to a Self-Hosted one when upgrading if --feature-gates=SelfHosting=true is set (which is the default)
[upgrade/health] If you strictly want to continue using a Static Pod-hosted control plane, set --feature-gates=SelfHosting=true when running 'kubeadm upgrade apply'
[upgrade/health] Checking Static Pod manifests exists on disk: All required Static Pod manifests exist on disk
[upgrade] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster (you can get this with 'kubectl -n kube-system get cm kubeadm-config -oyaml')
[upgrade/version] You have chosen to upgrade to version "v1.7.3"
[upgrade/versions] Cluster version: v1.7.1
[upgrade/versions] kubeadm version: v1.8.0-alpha.2.789+11f48dc291fe93
[upgrade/confirm] Are you sure you want to proceed with the upgrade? [y/N]: Y
[upgrade/prepull] Will prepull images for components [kube-apiserver kube-controller-manager kube-scheduler]
[upgrade/prepull] Prepulling image for component kube-scheduler.
[upgrade/prepull] Prepulling image for component kube-apiserver.
[upgrade/prepull] Prepulling image for component kube-controller-manager.
[upgrade/prepull] Prepulled image for component kube-scheduler.
[upgrade/prepull] Prepulled image for component kube-apiserver.
[upgrade/prepull] Prepulled image for component kube-controller-manager.
[upgrade/prepull] Successfully prepulled the images for all the control plane components
[upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.7.3"...
[upgrade/staticpods] Wrote upgraded Static Pod manifests to "/tmp/kubeadm-upgrade830923296"
[upgrade/staticpods] Moved upgraded manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backuped old manifest to "/tmp/kubeadm-upgrade830923296/old-manifests/kube-apiserver.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[apiclient] Found 1 Pods for label selector component=kube-apiserver
[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
[upgrade/staticpods] Moved upgraded manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backuped old manifest to "/tmp/kubeadm-upgrade830923296/old-manifests/kube-controller-manager.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[apiclient] Found 1 Pods for label selector component=kube-controller-manager
[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
[upgrade/staticpods] Moved upgraded manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backuped old manifest to "/tmp/kubeadm-upgrade830923296/old-manifests/kube-scheduler.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[apiclient] Found 1 Pods for label selector component=kube-scheduler
[apiclient] Found 0 Pods for label selector component=kube-scheduler
[apiclient] Found 1 Pods for label selector component=kube-scheduler
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
[self-hosted] Created TLS secret "ca" from ca.crt and ca.key
[self-hosted] Created TLS secret "apiserver" from apiserver.crt and apiserver.key
[self-hosted] Created TLS secret "apiserver-kubelet-client" from apiserver-kubelet-client.crt and apiserver-kubelet-client.key
[self-hosted] Created TLS secret "sa" from sa.pub and sa.key
[self-hosted] Created TLS secret "front-proxy-ca" from front-proxy-ca.crt and front-proxy-ca.key
[self-hosted] Created TLS secret "front-proxy-client" from front-proxy-client.crt and front-proxy-client.key
[self-hosted] Created secret "scheduler.conf"
[self-hosted] Created secret "controller-manager.conf"
[apiclient] Found 0 Pods for label selector k8s-app=self-hosted-kube-apiserver
[apiclient] Found 1 Pods for label selector k8s-app=self-hosted-kube-apiserver
[self-hosted] self-hosted kube-apiserver ready after 15.295571 seconds
[apiclient] Found 0 Pods for label selector k8s-app=self-hosted-kube-controller-manager
[apiclient] Found 1 Pods for label selector k8s-app=self-hosted-kube-controller-manager
[self-hosted] self-hosted kube-controller-manager ready after 15.804586 seconds
[apiclient] Found 0 Pods for label selector k8s-app=self-hosted-kube-scheduler
[apiclient] Found 1 Pods for label selector k8s-app=self-hosted-kube-scheduler
[self-hosted] self-hosted kube-scheduler ready after 18.958074 seconds
[apiconfig] Created RBAC rules
[addons] Applied essential addon: kube-proxy
[addons] Applied essential addon: kube-dns

PTAL and heads up @kubernetes/sig-cluster-lifecycle-pr-reviews @fabriziopandini @andrewrynhard @aaronlevy @ericchiang @mattmoyer @jamiehannaford @pipejakob @justinsb @jnardiello @craigtracey @kad @nckturner @idvoretskyi @calebamiles @jdumars

@timothysc

Automatic merge from submit-queue (batch tested with PRs 47416, 47408, 49697, 49860, 50162) kubeadm: Replace *clientset.Clientset with clientset.Interface **What this PR does / why we need it**: Needed for #48899 We should always use `clientset.Interface` instead of `*clientset.Clientset`, for better testability and all the other benefits of using an interface. **Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes # **Special notes for your reviewer**: Should be straightforward to merge **Release note**: ```release-note NONE ``` @timothysc @dmmcquay @pipejakob

@dmmcquay

Automatic merge from submit-queue (batch tested with PRs 50254, 50174, 50179) kubeadm: Add back labels for the Static Pod control plane **What this PR does / why we need it**: This Labels section has been removed now for a short period during the v1.8 dev cycle, but I found a good use-case for it; namely filtering Mirror Pods by the `component=kube-*` label when waiting for the self-hosted control plane to come up after an upgrade. It's not _really_ neccessary, but nice to have. Also noticed the lack of coverage for this func, so added a small unit test. **Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes # Dependency for: #48899 **Special notes for your reviewer**: **Release note**: ```release-note NONE ``` @kubernetes/sig-cluster-lifecycle-pr-reviews @dmmcquay @timothysc @mattmoyer

luxas · 2017-08-25T22:52:59Z

Removing the do-not-merge label as this is now ready for merge!! 🎉
Only has a small dep on #51130, but that doesn't mock much with this PR; can be LGTM'd now if you feel for that.

PTAL @kubernetes/sig-cluster-lifecycle-pr-reviews

timothysc

/lgtm due to time.

We need to get some e2e tests in place and talk with the release team on this.

timothysc · 2017-08-26T21:38:29Z

cmd/kubeadm/app/cmd/upgrade/apply.go

+
+// PerformStaticPodUpgrade performs the upgrade of the control plane components for a static pod hosted cluster
+func PerformStaticPodUpgrade(client clientset.Interface, waiter apiclient.Waiter, internalcfg *kubeadmapi.MasterConfiguration) error {
+	pathManager, err := upgrade.NewKubeStaticPodPathManagerUsingTempDirs(constants.GetStaticPodDirectory())


NewKubeStaticPods ... PathManagerUsingTempDirs seems unnecessary.

timothysc · 2017-08-26T21:39:16Z

cmd/kubeadm/app/cmd/upgrade/apply.go

+
+// PerformStaticPodUpgrade performs the upgrade of the control plane components for a static pod hosted cluster
+func PerformStaticPodUpgrade(client clientset.Interface, waiter apiclient.Waiter, internalcfg *kubeadmapi.MasterConfiguration) error {
+	pathManager, err := upgrade.NewKubeStaticPodPathManagerUsingTempDirs(constants.GetStaticPodDirectory())


For now just leave it.

k8s-github-robot · 2017-08-26T21:44:03Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: luxas, timothysc
We suggest the following additional approver: smarterclayton

Assign the PR to them by writing /assign @smarterclayton in a comment when ready.

No associated issue. Update pull-request body to add a reference to an issue, or get approval with /approve no-issue

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

luxas · 2017-08-27T16:37:35Z

fixed some golint failures

luxas · 2017-09-03T10:32:46Z

Rebased this now

/retest

kad · 2017-09-03T11:32:50Z

cmd/kubeadm/app/phases/upgrade/compute.go

+		//                                                                           v1.9.0-alpha.0                                             -> v1.9.0-alpha.1 -> v1.9.0-alpha.2
+
+		// Get and output the current latest unstable version
+		latestVersionStr, latestVersion, err := versionGetterImpl.VersionFromCILabel("latest", "experimental version")


Nit: I wouldn't really call "latest" as "experimental". In case of ci/* builds I agree to call them experimental, but alphas/betas already have some quality tags attached (even were not very high).

They are very experimental -- they are just cut from master without any special coordination that feature X, Y, Z must be there for it to ship.

kad · 2017-09-03T11:33:31Z

cmd/kubeadm/app/phases/upgrade/compute.go

+
+		minorUnstable := latestVersion.Components()[1]
+		// Get and output the current latest unstable version
+		previousBranch := fmt.Sprintf("latest-1.%d", minorUnstable-1)


We wouldn't hit it now, but for future (2.x versions), we probably want to have boundary check here.
Worth to add TODO in comments ?

this is in the queue already so I won't update it anymore.
I think this is probably the smallest problem we'll have regarding 2.x ;)

k8s-github-robot · 2017-09-03T12:01:09Z

/test all [submit-queue is verifying that this PR is safe to merge]

k8s-github-robot · 2017-09-03T12:48:22Z

Automatic merge from submit-queue

k8s-ci-robot · 2017-09-03T13:03:20Z

@luxas: The following tests failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
pull-kubernetes-bazel	`91be145`	link	`/test pull-kubernetes-bazel`
pull-kubernetes-e2e-kops-aws	`c575626`	link	`/test pull-kubernetes-e2e-kops-aws`
pull-kubernetes-e2e-gce-bazel	`c575626`	link	`/test pull-kubernetes-e2e-gce-bazel`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

jamiehannaford · 2017-09-04T08:04:25Z

@luxas Did we implement a solution for the discussion outlined in #48899 (comment)? If not, can we track this as a separate issue for 1.9?

luxas · 2017-09-04T08:09:31Z

@jamiehannaford please open a new issue for that

luxas added area/kubeadm sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. labels Jul 13, 2017

luxas assigned timothysc and mikedanese Jul 13, 2017

k8s-ci-robot added sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Jul 13, 2017

k8s-github-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. release-note Denotes a PR that will be considered when it comes time to generate release notes. labels Jul 13, 2017

timothysc added this to the v1.8 milestone Jul 17, 2017

k8s-github-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 17, 2017

timothysc suggested changes Jul 17, 2017

View reviewed changes

timothysc approved these changes Jul 17, 2017

View reviewed changes

k8s-ci-robot requested a review from jbeda July 18, 2017 19:04

luxas force-pushed the kubeadm_easy_upgrades_plan branch from 467b9dd to 3a8ee00 Compare July 26, 2017 20:47

k8s-github-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jul 26, 2017

luxas changed the title ~~Implement kubeadm upgrade plan~~ WIP: Implement the kubeadm upgrade command Jul 26, 2017

timothysc added the do-not-merge DEPRECATED. Indicates that a PR should not merge. Label can only be manually applied/removed. label Aug 2, 2017

luxas force-pushed the kubeadm_easy_upgrades_plan branch from 381c56f to 68395c5 Compare August 4, 2017 14:41

This was referenced Aug 4, 2017

kubeadm: Replace *clientset.Clientset with clientset.Interface #50162

Merged

kubeadm: Centralize commonly used paths/constants to the constants pkg #50173

Merged

kubeadm: Add back labels for the Static Pod control plane #50174

Merged

k8s-github-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 6, 2017

mhausenblas mentioned this pull request Aug 8, 2017

kubeadm upgrade docs kubernetes/website#4689

Closed

2 tasks

luxas force-pushed the kubeadm_easy_upgrades_plan branch from 9a08554 to f9e9135 Compare August 25, 2017 22:38

luxas mentioned this pull request Aug 25, 2017

Add the possibility to set return values for the FakeDiscovery implementation #51130

Merged

luxas removed the do-not-merge DEPRECATED. Indicates that a PR should not merge. Label can only be manually applied/removed. label Aug 25, 2017

timothysc approved these changes Aug 26, 2017

View reviewed changes

luxas added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 27, 2017

luxas force-pushed the kubeadm_easy_upgrades_plan branch from f5a359a to 91be145 Compare August 27, 2017 16:35

k8s-github-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 27, 2017

luxas added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 27, 2017

luxas force-pushed the kubeadm_easy_upgrades_plan branch from 91be145 to e6e8355 Compare September 3, 2017 08:03

k8s-github-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. and removed lgtm "Looks good to me", indicates that a PR is ready to be merged. labels Sep 3, 2017

luxas added 3 commits September 3, 2017 12:25

Fully implement the kubeadm upgrade functionality

c237ff5

Add unit tests for kubeadm upgrade

9498353

autogenerated bazel

c575626

luxas force-pushed the kubeadm_easy_upgrades_plan branch from e6e8355 to c575626 Compare September 3, 2017 09:29

luxas added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 3, 2017

kad reviewed Sep 3, 2017

View reviewed changes

k8s-github-robot merged commit b3efdeb into kubernetes:master Sep 3, 2017

jamiehannaford mentioned this pull request Sep 4, 2017

Investigate rollback options for upgrade kubernetes/kubeadm#420

Closed

chrisohaver mentioned this pull request Oct 24, 2017

core: migration from kube-dns coredns/coredns#1167

Closed

luxas mentioned this pull request Dec 25, 2017

Implement different verbosity levels in kubeadm kubernetes/kubeadm#340

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement the `kubeadm upgrade` command #48899

Implement the `kubeadm upgrade` command #48899

luxas commented Jul 13, 2017 •

edited

Loading

timothysc left a comment

timothysc Jul 17, 2017

timothysc commented Jul 18, 2017

luxas commented Jul 26, 2017

luxas commented Aug 4, 2017 •

edited

Loading

luxas commented Aug 25, 2017

timothysc left a comment

timothysc Aug 26, 2017

timothysc Aug 26, 2017

k8s-github-robot commented Aug 26, 2017

luxas commented Aug 27, 2017

luxas commented Sep 3, 2017

kad Sep 3, 2017

luxas Sep 3, 2017

kad Sep 3, 2017

luxas Sep 3, 2017

k8s-github-robot commented Sep 3, 2017

k8s-github-robot commented Sep 3, 2017

k8s-ci-robot commented Sep 3, 2017

jamiehannaford commented Sep 4, 2017

luxas commented Sep 4, 2017

Implement the kubeadm upgrade command #48899

Implement the kubeadm upgrade command #48899

Conversation

luxas commented Jul 13, 2017 • edited Loading

timothysc left a comment

Choose a reason for hiding this comment

timothysc Jul 17, 2017

Choose a reason for hiding this comment

timothysc commented Jul 18, 2017

luxas commented Jul 26, 2017

luxas commented Aug 4, 2017 • edited Loading

luxas commented Aug 25, 2017

timothysc left a comment

Choose a reason for hiding this comment

timothysc Aug 26, 2017

Choose a reason for hiding this comment

timothysc Aug 26, 2017

Choose a reason for hiding this comment

k8s-github-robot commented Aug 26, 2017

luxas commented Aug 27, 2017

luxas commented Sep 3, 2017

kad Sep 3, 2017

Choose a reason for hiding this comment

luxas Sep 3, 2017

Choose a reason for hiding this comment

kad Sep 3, 2017

Choose a reason for hiding this comment

luxas Sep 3, 2017

Choose a reason for hiding this comment

k8s-github-robot commented Sep 3, 2017

k8s-github-robot commented Sep 3, 2017

k8s-ci-robot commented Sep 3, 2017

jamiehannaford commented Sep 4, 2017

luxas commented Sep 4, 2017

Implement the `kubeadm upgrade` command #48899

Implement the `kubeadm upgrade` command #48899

luxas commented Jul 13, 2017 •

edited

Loading

luxas commented Aug 4, 2017 •

edited

Loading