Can you show your monitoring that shows that the pod exceeds its limit for an extended period of time (a few minutes?)

containers:
- name: busybox
image:
resources:
requests:
ephemeral-storage: "500Mi"
limits:
ephemeral-storage: "500Mi"

State: Running
Started: Mon, 10 Jun 2019 12:48:57 -0400
Ready: True
Restart Count: 0
Limits:
ephemeral-storage: 500Mi
Requests:
ephemeral-storage: 500Mi
Environment:

kubectl get po busybox-7cc68d968c-mb47z -n testns
NAME READY STATUS RESTARTS AGE
busybox-7cc68d968c-mb47z 1/1 Running 0 82m

kubectl exec -it busybox-7cc68d968c-mb47z -n testns -- bash
bash-4.2$ fallocate -l 2G /var/tmp/test2
bash-4.2$ du -sh /var/tmp/*
1.0G /var/tmp/test
2.0G /var/tmp/test2

bash-4.2$ exit

kubectl get po busybox-7cc68d968c-mb47z -n testns
NAME READY STATUS RESTARTS AGE
busybox-7cc68d968c-mb47z 1/1 Running 0 83m

I was able to recreate this issue on a 1.13.5 cluster

For the node that was being tested an EBS volume was attached to the instance and mounted as an xfs volume to /var/lib/docker

The deployment with resources set had a pod scheduled to the node in question.

"Execing" into the pod and running.

fallocate -l 2G /var/tmp/test1

Created a file larger than the set ephemeral storage limit of 500Mi. The pod was not evicted. Even waiting up to 10 minutes.

Starting again with a fresh volume and deployment.

Initially creating a 4G file within the new pod with an underlying volume of 5G mounted on /var/lib/docker

fallocate -l 4G /var/tmp/test1

caused imageGCManager to kick in due to the node DiskPressure condition rather than honouring ephemeral limits and evicting that one pod first

Jun 13 02:47:35 ip-10-0-2-15 kubelet[27133]: W0613 02:47:35.771392 27133 eviction_manager.go:333] eviction manager: attempting to reclaim ephemeral-storage
Jun 13 02:47:35 ip-10-0-2-15 kubelet[27133]: I0613 02:47:35.771424 27133 container_gc.go:85] attempting to delete unused containers
Jun 13 02:47:35 ip-10-0-2-15 kubelet[27133]: I0613 02:47:35.782369 27133 image_gc_manager.go:317] attempting to delete unused images
Jun 13 02:47:35 ip-10-0-2-15 kubelet[27133]: I0613 02:47:35.794272 27133 eviction_manager.go:344] eviction manager: must evict pod(s) to reclaim ephemeral-storage
Jun 13 02:47:35 ip-10-0-2-15 kubelet[27133]: I0613 02:47:35.794493 27133 eviction_manager.go:362] eviction manager: pods ranked for eviction: debug-887cd4775-2brw9_test(d9f7c5ee-8d84-11e9-b987-02f54a20dc4c), canal-js4tm_kube-system(53c05e18-8d66-11e9-b987-02f54a20dc4c), debug-887cd4775-fwvhq_test(a7ac5932-8d84-11e9-b987-02f54a20dc4c), debug-887cd4775-r9zcc_test(0695b54b-8d85-11e9-b987-02f54a20dc4c), debug-887cd4775-l6kxx_test(d18c07b5-8d84-11e9-b987-02f54a20dc4c), debug-887cd4775-rvv

The ranking for eviction, however, was correct and the pod debug-887cd4775-2brw9 was the pod used to create the volume.

cc @kubernetes/sig-storage-bugs @jingxu97

@pickledrick @arunbpt7 Could you please share your pod yaml file? You can also email me jinxu at google.com if you prefer. Thanks!

@jingxu97

apiVersion: apps/v1
kind: Deployment
metadata:
name: busybox
spec:
replicas: 1
selector:
matchLabels:
app: busybox
template:
metadata:
labels:
app: busybox
spec:
securityContext:
runAsUser: 99
fsGroup: 99
containers:
- name: busybox
image:
resources:
requests:
ephemeral-storage: “500Mi”
limits:
ephemeral-storage: “500Mi”

@arunbpt7 did you miss some part of yaml file?

@jingxu97

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: debug
  name: debug
spec:
  selector:
    matchLabels:
      app: debug
  template:
    metadata:
      labels:
        app: debug
    spec:
      containers:
      - image: quay.io/pickledrick/debug
        imagePullPolicy: Always
        name: debug
        resources:
          limits:
            ephemeral-storage: 500Mi
          requests:
            ephemeral-storage: 500Mi

@arunbpt7 can you query the summary api (localhost:10255/stats/summary) from the node that pod is running on to make sure it is measuring disk space correctly?

Our tests for this are not super consistent: https://k8s-testgrid.appspot.com/sig-node-kubelet#node-kubelet-serial&include-filter-by-regex=LocalStorageCapacityIsolationEviction, but are mostly green. I'll try and bump the timeout on the serial tests to see if we can get a clearer signal.

@dashpole

curl -s http://localhost:10255/stats/summary

the curl -s http://localhost:10255/stats/summary ran on the node where the pod is running and shows nothing .

It sounds like that is probably your problem then. If you don't have any metrics, the kubelet can't do its monitoring or eviction. Can you share your kubelet logs, or see if there are any errors related to metrics?

@pickledrick , can you share the kubelet logs

Hi all,

The insecure stats API appears to be deprecated.

#59666
kubernetes/kubeadm#732

I'm not able to reproduce it with 1.13.5 cluster, I'm having the following

debug-887cd4775-ckp4l   0/1     Evicted   0          12m
debug-887cd4775-hlvxw   1/1     Running   0          112s

@pickledrick - if you have access to generated certs you can use the secure port

Hi @yastij I'm not able to reproduce. It seems in my environment Evictions are happening eventually. I am sourcing more information to see if there is something else set in original reporters configuration.

@pickledrick

/var/lib/docker is a separate file system apart from node root fs

Hi @arunbpt7

Yes, My test environment reflects this. Are you able to confirm the docker version you are using in this environment?

Ran into something similar to this as well, but on a 1.12.8 cluster and using emptyDir. Initially the pod never got evicted, however in subsequent runs evictions worked properly and I haven't run into this since.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: storage-test
  namespace: storage-test
spec:
  template:
    metadata:
      labels:
        app: storage-test
    spec:
      containers:
      - name: storage-test-container
        image: k8s.gcr.io/ubuntu-slim:0.1
        resources:
          requests:
            ephemeral-storage: 10Mi
          limits:
            ephemeral-storage: 10Mi
        command: ["/bin/sh"]
        args: ["-c", "dd if=/dev/urandom of=/cache/file.txt count=100 bs=1048576; sleep 1h"]
        volumeMounts:
        - mountPath: "/cache"
          name: cache-volume
      volumes:
      - name: cache-volume
        emptyDir: {}

This happens with the writable layer when the runtime directory (e. g. /var/lib/crio, /var/lib/docker) is not on the kubernetes root filesystem; it's due to this code in the eviction manager.

It's not apparent to me why this was done, and the author of the code (in changeset 27901ad) appears to have moved on. I'm planning to open a PR on it.

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

/remove-lifecycle stale

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

@fejta-bot: Closing this issue.

hello

i got the same issue on K8S 1.16 and my docker image is 200GB . i have plenty of space in /var/lib/docker (500 GB out 450 GB Free) and i am getting this error. "ephermeral storage " ..can someone tell me what should be the fix.

i have this error : The node was low on resource: ephemeral-storage. Container k8stst was using 112619704Ki, which exceeds its request of 0

hm maybe not 100% related to this issue but I had a problem which was due to the fact that I had 2 filesystems. K8S couldn't manage that fact. See kubernetes/enhancements#361 (comment).

So I ended up to mount the filesystem and made docker and kubelet use the same partition with symlinks which solved the issue.