Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ImageGCFailed attempting to remove images in use by running containers #57006

Closed
jhorwit2 opened this issue Dec 10, 2017 · 3 comments · Fixed by #57020
Closed

ImageGCFailed attempting to remove images in use by running containers #57006

jhorwit2 opened this issue Dec 10, 2017 · 3 comments · Fixed by #57020
Assignees
@dixudx
Labels
kind/bug Categorizes issue or PR as related to a bug. sig/node Categorizes an issue or PR as relevant to SIG Node.

Comments

@jhorwit2
Copy link
Contributor

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug
/sig node

What happened:

The kubelet is attempting to GC images that are in use by running containers that the kubelet manages (we have no running containers on a host outside of k8s control).

  Warning  ImageGCFailed            20m (x508 over 1d)  kubelet, host  (combined from similar events): wanted to free 25419897241 bytes, but freed 0 bytes space with errors in image deletion: rpc error: code = 2 desc = Error response from daemon: {"message":"conflict: unable to delete 99e59f495ffa (cannot be forced) - image is being used by running container aec6e63fe827"}
  Warning  ImageGCFailed            11m                 kubelet, host  wanted to free 24121289113 bytes, but freed 0 bytes space with errors in image deletion: rpc error: code = 2 desc = Error response from daemon: {"message":"conflict: unable to delete 99e59f495ffa (cannot be forced) - image is being used by running container 2c6622239d05"}
  Warning  ImageGCFailed            6m                  kubelet, host  wanted to free 23035570585 bytes, but freed 0 bytes space with errors in image deletion: rpc error: code = 2 desc = Error response from daemon: {"message":"conflict: unable to delete 99e59f495ffa (cannot be forced) - image is being used by running container b65a78dd8f12"}
  Warning  ImageGCFailed            1m                  kubelet, host  wanted to free 22244608409 bytes, but freed 0 bytes space with errors in image deletion: rpc error: code = 2 desc = Error response from daemon: {"message":"conflict: unable to delete 99e59f495ffa (cannot be forced) - image is being used by running container 50f53d79ea51"}

What you expected to happen:

I expect the kubelet to not attempt to remove images being used by running containers or at least not spam events for this.

How to reproduce it (as minimally and precisely as possible):

I think the way to reproduce this is to have your kubelet reach eviction thresholds because that's how i noticed it.

Anything else we need to know?:

Environment:

  • Kubernetes version (use kubectl version): The kubelet is v1.7.6
@k8s-ci-robot k8s-ci-robot added kind/bug Categorizes issue or PR as related to a bug. sig/node Categorizes an issue or PR as relevant to SIG Node. labels Dec 10, 2017
@jhorwit2
Copy link
Contributor Author

It seems the image it's trying to delete is the pause container in use by pods running in the host network.

@huangjiasingle
Copy link

@jhorwit2 it's also hapened in my k8s cluster of v1.7.6

@dixudx dixudx mentioned this issue Dec 11, 2017
Merged
@dixudx
Copy link
Member

dixudx commented Dec 11, 2017

/assign

@dashpole dashpole mentioned this issue Jan 16, 2018
Closed
@dixudx dixudx mentioned this issue Jan 22, 2018
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dixudx dixudx

Labels
kind/bug Categorizes issue or PR as related to a bug. sig/node Categorizes an issue or PR as relevant to SIG Node.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

ignore images in used by running containers when GC dixudx/kubernetes
4 participants
@jhorwit2 @dixudx @huangjiasingle @k8s-ci-robot