Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

node not exist failure during node status update flush controller's log #30898

Closed
jingxu97 opened this issue Aug 18, 2016 · 4 comments
Closed

node not exist failure during node status update flush controller's log #30898

jingxu97 opened this issue Aug 18, 2016 · 4 comments
Assignees
@jingxu97
Labels
area/kubelet priority/backlog Higher priority than priority/awaiting-more-evidence. sig/storage Categorizes an issue or PR as relevant to SIG Storage.
Milestone
v1.7

Comments

@jingxu97
Copy link
Contributor

@saad-ali @matchstick
From issue #29903, I noticed that under attachdetach controller, reconciler keeps updating node status every 100ms. In some cases, the node no longer exists but reconciler still has a stale cache information of it. The error message quickly flush the kube-controller-manager's log.

  1. Quick fix: Lower the level of this log so by default it won't show up
  2. Right fix: have a good comprehensive understanding of all scenarios especially when certain components in the system fail/exit, how the volume manager should react/recover from them.

Related code and PR: https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/volume/attachdetach/reconciler/reconciler.go#L104
#29358
#30737
@jingxu97 jingxu97 added the sig/storage Categorizes an issue or PR as relevant to SIG Storage. label Aug 18, 2016
@jingxu97 jingxu97 added this to the v1.4 milestone Aug 18, 2016
@jingxu97 jingxu97 self-assigned this Aug 18, 2016
@saad-ali
Copy link
Member

Lowering the priority is non-ideal but should be fine for now

@jingxu97 jingxu97 changed the title node status failed log flush kube-controller-manager.log if node no longer exist node not exist failure during node status update should not block other nodes update Aug 18, 2016
@jingxu97 jingxu97 changed the title node not exist failure during node status update should not block other nodes update node not exist failure during node status update flush controller's log Aug 18, 2016
@jingxu97 jingxu97 mentioned this issue Aug 18, 2016
Merged
@goltermann goltermann added the priority/backlog Higher priority than priority/awaiting-more-evidence. label Sep 6, 2016
@goltermann goltermann modified the milestones: v1.5, v1.4 Sep 6, 2016
@dims
Copy link
Member

dims commented Nov 15, 2016

switching this to 1.6 as it's too late for 1.5. ok? (please switch it right back if you disagree)

@dims dims modified the milestones: v1.6, v1.5 Nov 15, 2016
@ethernetdan
Copy link
Contributor

Moving to 1.7 as late to happen in 1.6. Feel free to switch back if this is incorrect.

@ethernetdan ethernetdan modified the milestones: v1.7, v1.6 Mar 13, 2017
@saad-ali
Copy link
Member

saad-ali commented Jun 7, 2017

Fixed by PRs for k8s 1.7:

@saad-ali saad-ali closed this as completed Jun 7, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jingxu97 jingxu97

Labels
area/kubelet priority/backlog Higher priority than priority/awaiting-more-evidence. sig/storage Categorizes an issue or PR as relevant to SIG Storage.
Projects
None yet
Milestone
v1.7
Development

No branches or pull requests
6 participants
@dims @ethernetdan @goltermann @saad-ali @k8s-github-robot @jingxu97