Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Restarting kubelet with new node labels doesn't update labels for node #28051

Closed
guoshimin opened this issue Jun 24, 2016 · 17 comments
Closed

Restarting kubelet with new node labels doesn't update labels for node #28051

guoshimin opened this issue Jun 24, 2016 · 17 comments
Assignees
Labels
kind/bug Categorizes issue or PR as related to a bug. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/node Categorizes an issue or PR as relevant to SIG Node.

Comments

@guoshimin
Copy link
Contributor

kubelet version: 1.2.4

Steps to reproduce:

  • start kubelet with some labels using the --node-labels flag
  • wait for the node to register itself with the master
  • restart kubelet with a different set of labels

Expected result:
kubectl get no --show-labels shows the node with the updated labels

Actual result:
kubectl get no --show-labels shows the node with the original labels

@adohe-zz
Copy link

@guoshimin this is an alpha feature, and may has problem. More specifically, when we register node in kubelet, we always use Create, maybe we should use UpdateOrCreate here. @kubernetes/sig-node

@girishkalele girishkalele added the sig/node Categorizes an issue or PR as relevant to SIG Node. label Jun 26, 2016
@vishh
Copy link
Contributor

vishh commented Jun 27, 2016

This is a known issue and that's why the feature is still alpha. There is no concept of label ownership as of now since labels can be added to the node object from multiple sources.
We need to update kubelet to checkpoint the flags that it currently owns and use that knowledge while updating flags in the future.

A possible workaround meanwhile will be that of having kubelet delete existing flags and adding the new flags specified via --node-labels flags everytime it restarts.

@vishh vishh added priority/backlog Higher priority than priority/awaiting-more-evidence. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. and removed priority/backlog Higher priority than priority/awaiting-more-evidence. labels Jun 27, 2016
@vishh vishh added this to the v1.4 milestone Jun 27, 2016
@ncdc
Copy link
Member

ncdc commented Jul 1, 2016

Related #25811

@timothysc
Copy link
Member

@vishh do you know if there is an assignee for for 1.4. Our ops folks are not fond of having to manage node configs in maintain label sanity across restarts.

/cc @twiest @scollier

@vishh
Copy link
Contributor

vishh commented Jul 26, 2016

@timothysc No one is working on completing node labels AFAIK.

@vishh
Copy link
Contributor

vishh commented Jul 26, 2016

cc @dchen1107

@adohe-zz
Copy link

I would do this if we decide to persist node labels at last.

@goltermann goltermann modified the milestones: v1.5, v1.4 Sep 6, 2016
@dims
Copy link
Member

dims commented Nov 15, 2016

switching this to 1.6 as it's too late for 1.5. ok? (please switch it right back if you disagree)

@dims dims modified the milestones: v1.6, v1.5 Nov 15, 2016
@adohe-zz
Copy link

@dims I think we still don't make a final decision.

@dims dims modified the milestones: v1.5, v1.6 Nov 16, 2016
@dims
Copy link
Member

dims commented Nov 16, 2016

Ack @adohe please adjust milestone when appropriate

@saad-ali
Copy link
Member

@adohe Can you please triage this issue by EOD Friday. Is this a 1.5 issue or can it be punted to 1.6? If it is a 1.5 issue is it a blocker for the release (meaning release must wait for this to be fixed), or a non-blocker?

@euank
Copy link
Contributor

euank commented Nov 18, 2016

I don't think this can be a release blocker since it's not a bug. This feature functions as documented. fixing this issue is an additional feature, not a bugfix.

This can't break customers since it's been as-is (and as-documented) for multiple releases now.

I see no reason this additional feature (which requires additional kubelet persistence and has zero work thusfar) should go in post-feature freeze and shouldn't be targeted at 1.6.

Happy to be corrected if I'm misunderstanding anything

@dims
Copy link
Member

dims commented Nov 18, 2016

excellent @euank punting it to 1.6 :)

@dims dims modified the milestones: v1.6, v1.5 Nov 18, 2016
@timothysc timothysc added the kind/bug Categorizes issue or PR as related to a bug. label Nov 18, 2016
@resouer resouer self-assigned this Jan 2, 2017
@resouer
Copy link
Contributor

resouer commented Feb 8, 2017

Relies on #29459

@pwittrock
Copy link
Member

@resouer What is the status of this for 1.6?

@dchen1107
Copy link
Member

This is not targeted for 1.6 at all, and no assignee. We need to first finish #29459 for kubelet config in general, then come back with the node labels.

@resouer
Copy link
Contributor

resouer commented Sep 3, 2017

I've confirmed this has been fixed #46254

@resouer resouer closed this as completed Sep 3, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/node Categorizes an issue or PR as relevant to SIG Node.
Projects
None yet
Development

Successfully merging a pull request may close this issue.