/cc @vishh @ronnielai - per our chat today.

A few things I am trying to reason through:

1. we have no way of knowing the number of inodes reclaimed in response to image gc.
2. i am not sure if i am double counting inode consumption, more eyes appreciated.

The issue with (1) is that we may have actually reclaimed enough inodes to not need to kill a pod.

I am double counting pod inode consumption, but its double counting consistently (logs and rootfs), so I think it doesn't make a material impact.

If inode consumption is double counted, eviction-minimum-reclaim calculation will get impacted, won't it?

@ronnielai - the double counting is happening when calculating per pod usage primarily between the rootfs and logfs. in this case, the counting is then used to choose the greediest consumer, but as long as the counting is consistent, we will choose the greediest. for min-reclaim, that is evaluated relative to the actual rootfs or imagefs inodesFree value, so no double counting is there. i guess what i am trying to figure is if there is any benefit to counting per pod inodes for anything that is not the rootfs. thoughts?

Please add inode testing in TestMakeSignalObservations() and TestThresholdsMet().

LGTM with nits

I was experimenting with this more, and realized the following:

## HOST MACHINE
$ df -i
Filesystem                Inodes  IUsed    IFree IUse% Mounted on
/dev/mapper/fedora-root  3276800 714759  2562041   22% /
...
## INSIDE CONTAINER (overlay driver)
$ df -i       
Filesystem               Inodes  IUsed   IFree IUse% Mounted on
overlay                 3276800 714759 2562041   22% /
tmpfs                   1498577     18 1498559    1% /dev
tmpfs                   1498577     16 1498561    1% /sys/fs/cgroup
/dev/mapper/fedora-root 3276800 714759 2562041   22% /etc/hosts
shm                     1498577      1 1498576    1% /dev/shm
tmpfs                   1498577      9 1498568    1% /run/secrets/kubernetes.io/serviceaccount

Notice that the / inode consumption for the container matches the host. This means if I have multiple containers on a node, trying to rank pods by greediest consumer of inodes does not work with existing cAdvisor support. In practice, our pods will be ranked by QOS only unless we had a way of knowing the number of inodes unique to docker image + COW layer per container.

GIven that disk is best-effort in 1.4, are we ok with that behavior until we figure out a way to return meaningful values? Given that inodes consumption is independent of disk usage (i.e. touch foo.txt uses as much as 15GB file), it seems ranking pods by core QOS measures may be optimal behavior for now?

/cc @vishh @ronnielai

Unless we want to run find / -type f | wc -l inside the container fs which is just awful ;-)

I think we can put that as a limitation in https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/kubelet-eviction.md for now. @vishh, what do you think?

I'm totally fine with just using QoS for inodes exhaustion. Given that tracking inodes is not possible across all storage drivers, let's not do it for now.

@derekwaynecarr we can have cadvisor track inodes similar to how it tracks per-container disk usage.

I will update this PR and poke when ready.

@vishh @ronnielai - updated. right now, left place-holder to report 0 for inode_usage per container to make it a one-liner update when we get per container stats in cadvisor. added test cases. should be good to go.

Excepting the pending bytes comment, this pr lgtm.

rebased, fixed the byte lingering comment, tagging pr for merge.

@k8s-bot test this issue #30462

Added a commit now that we pass golint to get past jenkins verification error. Retagging.

GCE e2e build/test passed for commit 8261520.

• Test Results
• Build Log
• Test Artifacts
• Internal Jenkins Results

@k8s-bot test this [submit-queue is verifying that this PR is safe to merge]

GCE e2e build/test passed for commit 8261520.

• Test Results
• Build Log
• Test Artifacts
• Internal Jenkins Results

Automatic merge from submit-queue

㊗️ @derekwaynecarr & @ronnielai

#21546