Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kube-addon-updater.sh not working properly for Deployments #23471

Closed
fgrzadkowski opened this issue Mar 25, 2016 · 14 comments
Closed

kube-addon-updater.sh not working properly for Deployments #23471

fgrzadkowski opened this issue Mar 25, 2016 · 14 comments
Assignees
Labels
kind/bug Categorizes issue or PR as related to a bug. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle.

Comments

@fgrzadkowski
Copy link
Contributor

Current logic in kube-addon-updater.sh is the following:

  1. Check object names in the yaml files on disk and compare with objects with kubernetes.io/cluster-service=true label
  2. Delete objects not listed on disk which exist in the system
  3. Create not existing objects
  4. Delete&Create objects for which are using name suffixes to find next version (e.g. heapster-1.0 and heapster-1.1 with - used as a separator). This is currently done only for ReplicationControllers

Recently @Q-Lee in #22893 added support for Deployments in this script and removed name suffixes for objects using Deployments (i.e. heapsteR). This has several problems:

a) Without name change we will never update Deployment object
b) Once we add reconciling ReplicaSet object in addon updater it will clash with Deployment controller - controller will create ReplicaSet, which will be delete by addon updater as it's not explicitly listed in yaml files
c) Even if we fix addon-updater to simply compare object spec (so that we don't have to rely on object name changes) we have addon-resizer, that will modify object spec (request/limit fields) that will show as a difference when comparing spec and addon-updater will override it

I believe we should not cherry-pick this PR to 1.2 and rethink the whole approach.

@Q-Lee (PR author)
@mikedanese (PR reviewer)
@zmerlynn (understands kube-addon-updater.sh)
@bgrant0607 (reviews PRs for cherrypick)
@roberthbailey (active in discussion about this feature)

@fgrzadkowski fgrzadkowski added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. team/control-plane labels Mar 25, 2016
@fgrzadkowski
Copy link
Contributor Author

@piosz @mwielgus
Ref #23470

@bgrant0607
Copy link
Member

Process question:
It looks like this was LGTM with an unresolved question, and then was manually merged? Why the rush?

We should make kubectl apply work (probably needs a couple improvements, which we want anyway) and/or switch to Deployment Manager.

For (b), the kubernetes.io/cluster-service label should be removed from the pod template if that's used for reconciling.

What does the resizer base its decisions on? Anyway, we could probably just remove those values from the spec and use the LimitRange for the namespace.

@bgrant0607
Copy link
Member

Apply issues we'd probably want to fix: #19805, #19809, #22342, #17238, #16569

@bgrant0607
Copy link
Member

And cascading deletion should not be implemented in the client -- we plan to fix that in 1.3. #12143

@bgrant0607
Copy link
Member

Is there a reason we can't revert the PR for now?

@mikedanese
Copy link
Member

That was a pretty terrible review :(. Sorry about that.

a) Without name change we will never update Deployment object

The comment states that updates for non-renamed addons is not implemented. I missed that. I had initially recommended against the rename update for deployments since that's not the intended usage of deployments and I'm pretty sure it wouldn't work without modifying the addon updater. In the transition we would have two deployments fighting over the replicasets. Attempting to delete the old deployment would delete the new replicaset.

b) Once we add reconciling ReplicaSet object in addon updater it will clash with Deployment controller - controller will create ReplicaSet, which will be delete by addon updater as it's not explicitly listed in yaml files

Agree with Brian, only top level addon objects should carry the annotation.

c) Even if we fix addon-updater to simply compare object spec (so that we don't have to rely on object name changes) we have addon-resizer, that will modify object spec (request/limit fields) that will show as a difference when comparing spec and addon-updater will override it

We should modify an annotation when we want the deployment updater to reapply changes. We can compare the annotations, not the spec. That should solve this issue.

I will submit a revert.

@mikedanese
Copy link
Member

I'm pretty sure it wouldn't work without modifying the addon updater. In the transition we would have two deployments fighting over the replicasets. Attempting to delete the old deployment would delete the new replicaset.

This might work if we update the selector between deployment versions. It would make me sad that we would use deployment this way. Is this stuff required for 1.2?

@Q-Lee
Copy link
Contributor

Q-Lee commented Mar 25, 2016

@bgrant0607 sorry about the manual merge. I'm new to github, and thought the "merge" button was how we quicked the merge bot. I was unaware that "lgtm" meant auto-submit to our merge bot. I am unaware which question was unresolved, but it wouldn't greatly surprise me (a PR's flat comment space seems sub-optimal for multitasking).

@fgrzadkowski @mikedanese Why don't we just name the heapster deployment with a version in 1.2.x? The current use of deployments is to change resources, not software versions. When we want to support deployments properly for all addons, then we can remove the version from the name.

@mikedanese mikedanese added the kind/bug Categorizes issue or PR as related to a bug. label Mar 25, 2016
@Q-Lee
Copy link
Contributor

Q-Lee commented Mar 26, 2016

#23512

@roberthbailey
Copy link
Contributor

@Q-Lee was this fixed by #23512? If so, we should close it.

@bgrant0607
Copy link
Member

@roberthbailey No, I don't see that anything was done to fix this.

@mikedanese
Copy link
Member

I manually verified addon updates worked when I reviewed #23512. a, b, and c in the issue description are no longer a problem. This is the best we can do without making major changes to addon manager (#23233).

@Q-Lee
Copy link
Contributor

Q-Lee commented Apr 5, 2016

I left it open because kube-addon-updater.sh still doesn't behave how we want with Deployments. But the addon-resizer+heapster are currently working as Deployments in another way.

@mikedanese
Copy link
Member

That's a bullet in #23233 so we can track it there

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle.
Projects
None yet
Development

No branches or pull requests

5 participants