Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Node gc can't prune multi-repo images #70238

Closed
dbazhal opened this issue Oct 25, 2018 · 10 comments
Closed

Node gc can't prune multi-repo images #70238

dbazhal opened this issue Oct 25, 2018 · 10 comments
Labels
lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. sig/node Categorizes an issue or PR as relevant to SIG Node.

Comments

@dbazhal
Copy link

dbazhal commented Oct 25, 2018

What happened:
Image cleanup saves orphaned layers referenced in multiple repositories, which leads to node disk pressure.

What you expected to happen:
I expect that cleanup removes all untagged, unused images/layers making nodes healthy.

How to reproduce it (as minimally and precisely as possible):
Tag image twice in two different repositories in docker resigtry, use both images on the same kube node, drop it, watch logs(gc kubelet log).

Anything else we need to know?:

# kubelet gc log
unable to delete 176fc649a386 (must be forced) - image is referenced in multiple repositories

# docker images
docker images | grep -i 176fc649a386
docker-registry.default.svc:5000/demo-infra/showcase                           <none>              176fc649a386        38 hours ago        1.13 GB
docker-registry.default.svc:5000/demo-testing/showcase                         <none>              176fc649a386        38 hours ago        1.13 GB

Environment:

  • Kubernetes version - v1.11.0+d4cacc0
  • Cloud provider or hardware configuration: bare metal
  • OS (e.g. from /etc/os-release): centos7
  • Kernel (e.g. uname -a): 3.10.0-862.14.4.el7.x86_64
  • Install tools: openshift ansible installer
  • Others: docker version - 1.13.1, build 8633870/1.13.1

/king bug
@k8s-ci-robot k8s-ci-robot added the needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Oct 25, 2018
@dbazhal
Copy link
Author

dbazhal commented Oct 25, 2018

/sig node

@k8s-ci-robot k8s-ci-robot added sig/node Categorizes an issue or PR as relevant to SIG Node. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Oct 25, 2018
@flyingcougar
Copy link

flyingcougar commented Nov 20, 2018 •
edited
Loading

👍 I am facing the same issue.
#29316 Fixed multiple Image tags by checking imageInspect.RepoTags but i guess we need to check for imageInspect.RepoDigests as well

@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Feb 18, 2019
@dbazhal
Copy link
Author

dbazhal commented Feb 19, 2019

/remove-lifecycle stale

@k8s-ci-robot k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Feb 19, 2019
@myniva
Copy link

myniva commented Mar 14, 2019

Is this maybe fixed by #70647?

@dbazhal
Copy link
Author

dbazhal commented Mar 14, 2019

Don't know - too few details i know about both cases. Will test after release, and close if issue is outdated or collect more info.

@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jun 12, 2019
@fejta-bot
Copy link

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jul 12, 2019
@fejta-bot
Copy link

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

@k8s-ci-robot
Copy link
Contributor

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. sig/node Categorizes an issue or PR as relevant to SIG Node.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@myniva @dbazhal @flyingcougar @k8s-ci-robot @fejta-bot