Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't restore image if layer does not exist #36304

Merged
merged 1 commit into from
Feb 15, 2018

Conversation

lowenna
Copy link
Member

@lowenna lowenna commented Feb 13, 2018

Signed-off-by: John Howard jhoward@microsoft.com

This was reported internally to me, and now I've run into it myself. If the daemon is under debug mode and killed at the right time, it's possible to leave an image referring to a layer which doesn't exist. During subsequent daemon startup, the daemon terminates and fails to start during image store restore() saying layer does not exist.

This fix is similar to the check slightly higher in the restore() function which doesn't error out if is.Get() fails, instead logging it, but now checking explicitly for ErrLayerDoesNotExist when getting the layer from the layerstore.

Signed-off-by: John Howard <jhoward@microsoft.com>
Copy link
Member

@dnephin dnephin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've hit this problem a few times myself

LGTM

Copy link
Member

@thaJeztah thaJeztah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@thaJeztah
Copy link
Member

ping @tonistiigi PTAL

@tonistiigi
Copy link
Member

This seems ok to fix the broken state on disk but we should look into the actual problem as well instead of just ignoring the error. Have we looked into why this problem is appearing? Looking at

moby/image/store.go

Lines 247 to 249 in 1474ec1

if imageMeta.layer != nil {
return is.lss[img.OperatingSystem()].Release(imageMeta.layer)
}
the order of releasing layers seems to be correct. The image is always deleted before the layers are deleted so it shouldn't leave broken layers.

@dnephin
Copy link
Member

dnephin commented Feb 15, 2018

In my case, I have only ever hit this while developing new features that were broken when I ran a test daemon, and left images without the layers being cleaned up. I've never seen it in a released version or something in master.

@lowenna
Copy link
Member Author

lowenna commented Feb 15, 2018

@tonistiigi The two cases where it's been hit here have both been similar to @dnephin while killing/stopping a daemon while under a debugger, and during development. I don't think there's a real-world scenario.

@dnephin dnephin merged commit b1a1234 into moby:master Feb 15, 2018
@lowenna lowenna deleted the jjh/dontrestoreimageformissinglayer branch February 15, 2018 19:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants