-
Notifications
You must be signed in to change notification settings - Fork 40.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Eviction manager evicts based on inode consumption #35137
Eviction manager evicts based on inode consumption #35137
Conversation
@dashpole can you add a test case that evicts based on inodes similar to the other test cases that work on disk? |
yes, the test is going out in a seperate PR #33955. I will be testing 3 pods: one creates empty files in it's container, one creates empty files in an emptydir volume, and another creates one, large file. The eviction manager should hit inodepressure and evict both of the containers making empty files. I still need to look into making the test skip the "empty files in container" pod when using devicemapper. I also want to run the test more, to make sure it isnt flaky before adding the test. |
@dashpole -- the e2e test is good, but we should also be able to mock the test in unit testing as well via something like the following: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/eviction/eviction_manager_test.go#L315 |
@derekwaynecarr ill add a unit test to this PR |
ac8d817
to
c465205
Compare
Jenkins GCE etcd3 e2e failed for commit c4652055208f3751baa41dd69c0139a8ca49194d. Full PR test history. The magic incantation to run this job again is |
Jenkins GCI GCE e2e failed for commit c4652055208f3751baa41dd69c0139a8ca49194d. Full PR test history. The magic incantation to run this job again is |
Jenkins GCE e2e failed for commit c4652055208f3751baa41dd69c0139a8ca49194d. Full PR test history. The magic incantation to run this job again is |
Jenkins GKE smoke e2e failed for commit c4652055208f3751baa41dd69c0139a8ca49194d. Full PR test history. The magic incantation to run this job again is |
Jenkins Kubemark GCE e2e failed for commit c4652055208f3751baa41dd69c0139a8ca49194d. Full PR test history. The magic incantation to run this job again is |
I found a bug in the eviction manager tests. We were not assigning pods a UID, so the cached stats function was always returning the last (in order of when pods were added) pod's statsFunc. This meant that all calls to statsFunc returned the same value. This did not break tests because we always passed in the pod we wanted to fail as the first pod, and the eviction manager would leave that ordering intact when pods tie in their consumption. Changing the order of pods broke all tests in eviction_manager_test.go. Changing the newPod function in helpers_test.go to include a UID based on the name fixed the problem. |
Jenkins verification failed for commit c4652055208f3751baa41dd69c0139a8ca49194d. Full PR test history. The magic incantation to run this job again is |
c465205
to
f908781
Compare
Jenkins GCE Node e2e failed for commit f90878158b4670dc6b393af63358843e7dcc437c. Full PR test history. The magic incantation to run this job again is |
f908781
to
0845625
Compare
@derekwaynecarr @dchen1107 this all ready to go as soon as one of you has the chance to review it. |
pkg/kubelet/eviction/eviction_manager_test.go, line 103 at r1 (raw file):
Changed ordering to prevent regression on bug described in git comments. Comments from Reviewable |
pkg/kubelet/eviction/eviction_manager_test.go, line 117 at r1 (raw file):
Eviction manager reorders pods when choosing which pod to evict, so grab a reference to the correct pod to evict now... Comments from Reviewable |
pkg/kubelet/eviction/helpers_test.go, line 1570 at r1 (raw file):
This is to fix the bug described in the git comments. Comments from Reviewable |
/lgtm |
pkg/kubelet/server/stats/summary.go, line 118 at r1 (raw file):
Is this safe to do? Comments from Reviewable |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i missed this originally, thx for pointer.
@@ -115,6 +115,8 @@ func (sb *summaryBuilder) build() (*stats.Summary, error) { | |||
return nil, fmt.Errorf("Missing stats for root container") | |||
} | |||
|
|||
nodeFsInodesUsed := *sb.rootFsInfo.Inodes - *sb.rootFsInfo.InodesFree |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
you will need to check for nil on both values. for imagefs, it will be nil for both inodes and inodesfree values.
0845625
to
4ca7f9f
Compare
pkg/kubelet/server/stats/summary.go, line 118 at r1 (raw file):
|
@k8s-bot gci gke e2e test this |
@k8s-bot gci gke e2e test this |
Jenkins GCI GKE smoke e2e failed for commit 4ca7f9f. Full PR test history. The magic incantation to run this job again is |
@k8s-bot gci gke e2e test this |
LGTM |
@k8s-bot unit test this |
Jenkins unit/integration failed for commit 4ca7f9f. Full PR test history. The magic incantation to run this job again is |
@k8s-bot unit test this |
Automatic merge from submit-queue |
Automatic merge from submit-queue Per Volume Inode Accounting Collects volume inode stats using the same find command as cadvisor. The command is "find _path_ -xdev -printf '.' | wc -c". The output is passed to the summary api, and will be consumed by the eviction manager. This cannot be merged yet, as it depends on changes adding the InodesUsed field to the summary api, and the eviction manager consuming this. Expect tests to fail until this happens. DEPENDS ON #35137
Fixes: #32526 Integrate Cadvisor per-container inode stats into the summary api. Make the eviction manager act based on inode consumption to evict pods using the most inodes.
This PR is pending on a cadvisor godeps update which will be included in PR #35136
This change is