Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cinder volume can't be unmounted when pod is deleted #24717

Closed
chengyli opened this issue Apr 24, 2016 · 1 comment · Fixed by #24718
Closed

cinder volume can't be unmounted when pod is deleted #24717

chengyli opened this issue Apr 24, 2016 · 1 comment · Fixed by #24718
Labels
area/kubelet sig/node Categorizes an issue or PR as relevant to SIG Node.

Comments

@chengyli
Copy link
Contributor

We noticed that if kubelet's root-dir is a symlink, when pod with a cinder volume is deleted, cinder volume is still mounted even though pod's containers have been removed. The below errors are seen in kubelet log.

Apr 23 17:11:34 kubernetes-minion-1-3169 kubelet[31412]: W0423 17:11:34.263560   31412 mount.go:99] could not determine device for path: "/mnt/kubelet/pods/473b273e-0976-11e6-851a-74dbd1a0bde4/volumes/kubernetes.io~cinder/pv-openshift-registry"
Apr 23 17:11:34 kubernetes-minion-1-3169 kubelet[31412]: I0423 17:11:34.263570   31412 cinder.go:354] Directory /mnt/kubelet/pods/473b273e-0976-11e6-851a-74dbd1a0bde4/volumes/kubernetes.io~cinder/pv-openshift-registry is not mounted
Apr 23 17:11:34 kubernetes-minion-1-3169 kubelet[31412]: E0423 17:11:34.263578   31412 kubelet.go:1990] Could not tear down volume "473b273e-0976-11e6-851a-74dbd1a0bde4/pv-openshift-registry": directory /mnt/kubelet/pods/473b273e-0976-11e6-851a-74dbd1a0bde4/volumes/kubernetes.io~cinder/pv-openshift-registry is not mounted
Apr 23 17:11:34 kubernetes-minion-1-3169 kubelet[31412]: I0423 17:11:34.263688   31412 kubelet.go:1901] Orphaned pod "473b273e-0976-11e6-851a-74dbd1a0bde4" found, but volumes are not cleaned up; err: <nil>, volumes: [0xc2082bede0]
@chengyli
Copy link
Contributor Author

for example, kubelet's root-dir is /mnt/kubelet, but /mnt is a symlink to /var/mnt. So the Linux /proc shows the volume is mounted under /var/mnt/kubelet/..., but mount point in kubelet is /mnt/kubelet.

-bash-4.2# cat /proc/mounts | grep vdc
/dev/vdc /var/mnt/kubelet/pods/d3aa9349-097d-11e6-851a-74dbd1a0bde4/volumes/kubernetes.io~cinder/pv-openshift-registry ext4 rw,relatime,data=ordered 0 0

Then the compare mps[i].Path == mountPath in kubelet code can never be true.

func GetMountRefs(mounter Interface, mountPath string) ([]string, error) {
    mps, err := mounter.List()
    if err != nil {
        return nil, err
    }

    // Find the device name.
    deviceName := ""
    for i := range mps {
        if mps[i].Path == mountPath {
            deviceName = mps[i].Device
            break
        }
    }

    // Find all references to the device.
    var refs []string
    if deviceName == "" {
        glog.Warningf("could not determine device for path: %q", mountPath)
    } else {
        ...

chengyli added a commit to chengyli/kubernetes that referenced this issue Apr 28, 2016
fix kubernetes#24717
If kubelet root-dir is a symlink, the pod's cinder volume dir can't be
umounted even after pod is deleted.
This patch reads target path of symlink before comparing with entries in
proc mounts.
@mml mml added area/kubelet sig/node Categorizes an issue or PR as relevant to SIG Node. labels May 2, 2016
roberthbailey added a commit that referenced this issue May 6, 2016
fix cinder volume dir umount issue #24717
derekwaynecarr pushed a commit to derekwaynecarr/kubernetes that referenced this issue May 10, 2016
fix kubernetes#24717
If kubelet root-dir is a symlink, the pod's cinder volume dir can't be
umounted even after pod is deleted.
This patch reads target path of symlink before comparing with entries in
proc mounts.
k8s-github-robot pushed a commit that referenced this issue Feb 18, 2017
Automatic merge from submit-queue (batch tested with PRs 41401, 41195, 41664, 41521, 41651)

Allow `make test` to pass on OSX

**What this PR does / why we need it**: `make test` doesn't pass on my OSX setup (10.11.6, go1.7, docker 1.13.1) on `master`, `release-1.5`, nor `release-1.4`.  Our [docs on unit tests](https://github.com/kubernetes/community/blob/master/contributors/devel/testing.md#unit-tests) say they should always pass on OS X.  This PR allows them to pass.

**Release note**:
```release-note
NONE
```

ref: #24717 for the motivation behind dereferencing mount symlinks

/cc @kubernetes/sig-testing-pr-reviews
wgliang pushed a commit to wgliang/utils that referenced this issue Jun 11, 2018
fix kubernetes/kubernetes#24717
If kubelet root-dir is a symlink, the pod's cinder volume dir can't be
umounted even after pod is deleted.
This patch reads target path of symlink before comparing with entries in
proc mounts.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/kubelet sig/node Categorizes an issue or PR as relevant to SIG Node.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants