Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fail container start if its requested device plugin resource is unknown. #67145

Merged
merged 1 commit into from
Aug 21, 2018

Conversation

jiayingz
Copy link
Contributor

@jiayingz jiayingz commented Aug 8, 2018

With the change, Kubelet device manager now checks whether it has cached option state for the requested device plugin resource to make sure the resource is in ready state when we start the container.

What this PR does / why we need it:

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #67107

Special notes for your reviewer:

Release note:

Fail container start if its requested device plugin resource hasn't registered after Kubelet restart.

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Aug 8, 2018
@jiayingz
Copy link
Contributor Author

jiayingz commented Aug 8, 2018

/sig node

@k8s-ci-robot k8s-ci-robot added the sig/node Categorizes an issue or PR as relevant to SIG Node. label Aug 8, 2018
@jiayingz
Copy link
Contributor Author

jiayingz commented Aug 8, 2018

/assign @vikaschoudhary16 @RenaudWasTaken

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 8, 2018
@RenaudWasTaken
Copy link
Contributor

Are failed container "retried" or is this up to another abstraction?

@jiayingz
Copy link
Contributor Author

jiayingz commented Aug 8, 2018

Kubelet will retry with exponential backoff latency.

have cached option state to make sure the device plugin resource is
in ready state when we start the container.
@vikaschoudhary16
Copy link
Contributor

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 21, 2018
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jiayingz, vikaschoudhary16

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-github-robot
Copy link

/test all

Tests are more than 96 hours old. Re-running tests.

@k8s-github-robot
Copy link

/test all [submit-queue is verifying that this PR is safe to merge]

@k8s-github-robot
Copy link

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here.

@k8s-github-robot k8s-github-robot merged commit d017beb into kubernetes:master Aug 21, 2018
@dashpole
Copy link
Contributor

This seems to have caused the device plugin tests to fail: https://k8s-testgrid.appspot.com/sig-node-kubelet#kubelet-serial-gce-e2e&include-filter-by-regex=DevicePlugin

@jiayingz
Copy link
Contributor Author

Ah right. I need to update the device plugin e2e_node test to cope with this behavior change. I will send a PR to fix that. @dashpole thanks a lot for the notification!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/node Categorizes an issue or PR as relevant to SIG Node. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.
Projects
None yet
6 participants