Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

respect ExecProbeTimeout=false for dockershim #100200

Merged
merged 1 commit into from
Apr 11, 2021
Merged

respect ExecProbeTimeout=false for dockershim #100200

merged 1 commit into from
Apr 11, 2021

Conversation

jackfrancis
Copy link
Contributor

What type of PR is this?

/kind bug

What this PR does / why we need it:

This PR ensures that the kubelet ExecProbeTimeout feature gate is respected when evaluating whether to timeout an exec probe's context when its configured timeoutSeconds period has expired.

The current implementation has the effect of ignoring the probe outcome after the timeoutSeconds period has expired.

Which issue(s) this PR fixes:

Fixes #100198

Special notes for your reviewer:

I've tested that a build with this PR restored the expected behavior, where timeouts are simply ignored.

The original PR that fixed the buggy timeout behavior does not reproduce this new "probes always succeed pathology" according to my tests:

#94115

Further testing confirmed that this commit is the first that reproduces this new bug:

#96495

It appears that this surface area change ommitted the ExecProbeTimeout feature gate:

https://github.com/kubernetes/kubernetes/pull/96495/files#diff-de4678da3e9d01e8ee0aadd641deb2ffec3a53c8603476509e59fa8dc5449deaL117

Does this PR introduce a user-facing change?

respect ExecProbeTimeout=false for dockershim

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/bug Categorizes issue or PR as related to a bug. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Mar 12, 2021
@k8s-ci-robot
Copy link
Contributor

Hi @jackfrancis. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Mar 12, 2021
@k8s-ci-robot k8s-ci-robot added area/kubelet sig/node Categorizes an issue or PR as relevant to SIG Node. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Mar 12, 2021
@jackfrancis
Copy link
Contributor Author

cc @andrewsykim @chewong @lachie83

It wasn't obvious to me how me might inject a feature gate config into the mocks used by TestExecInContainer in pkg/kubelet/dockershim/exec_test.go. If we can figure that out it should be easy to add a new test case to this PR that proves the fix.

Additionally, do you think it would make sense to include something like this in this PR?:

chewong@b59d903

@dims
Copy link
Member

dims commented Mar 13, 2021

/assign @ehashman @andrewsykim

@ehashman
Copy link
Member

/ok-to-test
/triage accepted

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Mar 13, 2021
@jackfrancis
Copy link
Contributor Author

/retest

@k8s-ci-robot k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Mar 15, 2021
@jackfrancis
Copy link
Contributor Author

/test pull-kubernetes-e2e-aks-engine-azure

@k8s-ci-robot k8s-ci-robot added area/test sig/testing Categorizes an issue or PR as relevant to SIG Testing. labels Mar 15, 2021
This was referenced Apr 9, 2021
Closed
Closed
ehashman
ehashman approved these changes Apr 9, 2021
View reviewed changes
Copy link
Member

@ehashman ehashman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dchen1107, ehashman, hasheddan, jackfrancis

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@bridgetkromhout
Copy link
Member

/retest

@ehashman
Copy link
Member

ehashman commented Apr 9, 2021

flake #100556

@jackfrancis
Copy link
Contributor Author

/retest

3 similar comments
@bridgetkromhout
Copy link
Member

/retest

@bridgetkromhout
Copy link
Member

/retest

@bridgetkromhout
Copy link
Member

/retest

@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

6 similar comments
@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Apr 10, 2021 •
edited
Loading

@jackfrancis: The following tests failed, say /retest to rerun all failed tests:

Test name Commit Details Rerun command
pull-kubernetes-e2e-aks-engine-azure d699cadf9cd531bd67e8410449c36856ece914c7 link /test pull-kubernetes-e2e-aks-engine-azure
pull-kubernetes-e2e-aks-engine-azure-windows d699cadf9cd531bd67e8410449c36856ece914c7 link /test pull-kubernetes-e2e-aks-engine-azure-windows

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@k8s-ci-robot k8s-ci-robot merged commit 3eac797 into kubernetes:master Apr 11, 2021
@k8s-ci-robot k8s-ci-robot added this to the v1.22 milestone Apr 11, 2021
@pacoxu pacoxu mentioned this pull request Apr 12, 2021
Merged
This was referenced Apr 12, 2021
Closed
Merged
Merged
k8s-ci-robot added a commit that referenced this pull request Apr 22, 2021
@k8s-ci-robot
Merge pull request #101127 from jackfrancis/automated-cherry-pick-of-…
d906e15 
…#100200-upstream-release-1.21

Automated cherry pick of #100200: respect ExecProbeTimeout and #101006: exec test should not run in Parallel
k8s-ci-robot added a commit that referenced this pull request Apr 22, 2021
@k8s-ci-robot
Merge pull request #101126 from jackfrancis/automated-cherry-pick-of-…
9b8b828 
…#100200-upstream-release-1.20

Automated cherry pick of #100200: respect ExecProbeTimeout and #101006: exec test should not run in Parallel
@jackfrancis jackfrancis deleted the ctx-respect-ExecProbeTimeout branch April 22, 2021 17:12
@kakarotbyte kakarotbyte mentioned this pull request Aug 26, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@andrewsykim andrewsykim andrewsykim left review comments

@chewong chewong chewong left review comments

@SergeyKanzhelev SergeyKanzhelev SergeyKanzhelev left review comments

@liggitt liggitt liggitt left review comments

@hasheddan hasheddan hasheddan approved these changes

@ehashman ehashman ehashman approved these changes

@odinuge odinuge Awaiting requested review from odinuge

@yujuhong yujuhong Awaiting requested review from yujuhong

Assignees

@ehashman ehashman

@SergeyKanzhelev SergeyKanzhelev

@andrewsykim andrewsykim

@hasheddan hasheddan

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/kubelet area/test cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/node Categorizes an issue or PR as relevant to SIG Node. sig/testing Categorizes an issue or PR as relevant to SIG Testing. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
SIG Node CI/Test Board
Archived in project
SIG Node: code and documentation PRs
Archived in project
Milestone
v1.22
Development

Successfully merging this pull request may close these issues.

exec probes that exceed timeout always succeed when using ExecProbeTimeout=false