AWS: Move enforcement of attached AWS device limit from kubelet to scheduler #23254
+10
−26
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
k8s-github-robot
added
the
size/S
|
Fixes #22994 |
|
@justinsb @kubernetes/rh-storage, please review. |
|
@jsafrane edit the original comment to include the |
| @@ -1050,19 +1050,20 @@ func (self *awsInstance) getMountDevice(volumeID string, assign bool) (assigned | |||
| return mountDevice(""), false, nil | |||
| } | |||
|
|
|||
| // Check all the valid mountpoints to see if any of them are free | |||
| valid := instanceType.getEBSMountDevices() | |||
There was a problem hiding this comment.
Is there another caller of getEBSMountDevices() or should you be deleting that code as well?
There was a problem hiding this comment.
Was going to say the same thing - it makes sense to inline it because the volume list is now pretty big, but do delete getEBSMountDevices
jsafrane
force-pushed
the
devel/ulimited-aws-devices
branch
from
c2e88c4 to
37b969c
Compare
k8s-github-robot
added
size/M
|
TravisCI error: Looks like flake. |
Limit of nr. of attached EBS volumes to a node is now enforced by scheduler. It
can be adjusted by KUBE_MAX_PD_VOLS env. variable there.
Therefore we don't need the same check in kubelet. If the system admin wants to
attach more, we should allow it.
Kubelet limit is now 650 attached volumes ('ba'..'zz').
jsafrane
force-pushed
the
devel/ulimited-aws-devices
branch
from
37b969c to
e4dc670
Compare
|
@justinsb, can you please take a look? |
justinsb
added
lgtm
justinsb
changed the title
|
Sorry for the delay @jsafrane . I changed the title to be a clearer (?) release note - feel free to tweak! |
|
@k8s-bot test this issue: #IGNORE |
|
GCE e2e build/test passed for commit e4dc670. |
|
Automatic merge from submit-queue |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Limit of nr. of attached EBS volumes to a node is now enforced by scheduler. It can be adjusted by
KUBE_MAX_PD_VOLSenv. variable there. Therefore we don't need the same check in kubelet. If the system admin wants to attach more, we should allow it.Kubelet limit is now 650 attached volumes ('ba'..'zz').
Note that the scheduler counts only pods assigned to a node. When a pod is deleted and a new pod is scheduled on a node, kubelet start (slowly) detaching the old volume and (slowly) attaching the new volume. Depending on AWS speed it may happen that more than KUBE_MAX_PD_VOLS volumes are actually attached to a node for some time! Kubelet will clean it up in few seconds / minutes (both attach/detach is quite slow).
Fixes #22994