Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

RemoveImage service failed - image is being used by running container #60670

Closed
dawidmalina opened this issue Mar 1, 2018 · 6 comments
Closed

RemoveImage service failed - image is being used by running container #60670

dawidmalina opened this issue Mar 1, 2018 · 6 comments
Labels
kind/bug Categorizes issue or PR as related to a bug. sig/node Categorizes an issue or PR as relevant to SIG Node.

Comments

@dawidmalina
Copy link

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug

/kind feature

What happened:
RemoveImage service is trying to remove pause image. My kubelet is configured with:

--pod-infra-container-image=docker-release/infra/pause:v3.0

I see in the logs that controller is trying to free up some space:

Mar  1 21:36:33 kubernetes-master-ubuntu kubelet[9576]: I0301 21:36:33.776112    9576 image_gc_manager.go:285] [imageGCManager]: Disk usage on image filesystem is at 76% which is over the high threshold (70%). Trying to free 17049925632 bytes
Mar  1 21:36:33 kubernetes-master-ubuntu kubelet[9576]: I0301 21:36:33.789323    9576 image_gc_manager.go:350] [imageGCManager]: Removing image "sha256:1e62b686f055cbaf2813102da095f7cee1e0805f7b4b124a6a7beaba06d07a3c" to free 840632 bytes

But in the kubelet logs i see error that service failed:

Mar  1 21:36:33 kubernetes-master-ubuntu kubelet[9576]: E0301 21:36:33.795829    9576 remote_image.go:130] RemoveImage "sha256:1e62b686f055cbaf2813102da095f7cee1e0805f7b4b124a6a7beaba06d07a3c" from image service failed: rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete 1e62b686f055 (cannot be forced) - image is being used by running container 5f79894ff5bd

Service is trying to remove image 1e62b686f055 but this is pause image:

docker images |grep 1e62b686f055
docker-release/infra/pause                                   v3.0                1e62b686f055        4 weeks ago         841kB

What you expected to happen:
Service should skip pause image and all images used (needed) by cluster components.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

  • Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.7", GitCommit:"b30876a5539f09684ff9fde266fda10b37738c9c", GitTreeState:"clean", BuildDate:"2018-01-16T21:59:57Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.7", GitCommit:"b30876a5539f09684ff9fde266fda10b37738c9c", GitTreeState:"clean", BuildDate:"2018-01-16T21:52:38Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

  • Cloud provider or hardware configuration:
    DinD test environment

  • OS (e.g. from /etc/os-release):

NAME="Ubuntu"
VERSION="16.04.4 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.4 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial
  • Kernel (e.g. uname -a):
Linux kubernetes-master-ubuntu 4.9.60-linuxkit-aufs #1 SMP Mon Nov 6 16:00:12 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
  • Install tools:
  • Others:
@k8s-ci-robot k8s-ci-robot added needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. kind/bug Categorizes issue or PR as related to a bug. labels Mar 1, 2018
@dawidmalina
Copy link
Author

/sig node

@k8s-ci-robot k8s-ci-robot added sig/node Categorizes an issue or PR as relevant to SIG Node. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Mar 1, 2018
@Pingan2017
Copy link
Member

resolved #57020 #58229

@vikaschoudhary16
Copy link
Contributor

Should this be closed?

@dawidmalina
Copy link
Author

@Pingan2017 what k8s version have this fix. I will try to test and validate?

@vikaschoudhary16
Copy link
Contributor

vikaschoudhary16 commented Mar 3, 2018 •
edited
Loading

@dawidmalina You can confirm on v1.10.0-beta.1

@dawidmalina
Copy link
Author

Ok. Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug. sig/node Categorizes an issue or PR as relevant to SIG Node.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@dawidmalina @vikaschoudhary16 @k8s-ci-robot @Pingan2017