Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tweak up flaking end-to-end tests of API Priority and Fairness #96874

Merged
merged 6 commits into from
Dec 4, 2020
Merged

Tweak up flaking end-to-end tests of API Priority and Fairness #96874

merged 6 commits into from
Dec 4, 2020

Conversation

MikeSpreitzer
Copy link
Member

@MikeSpreitzer MikeSpreitzer commented Nov 25, 2020 •
edited
Loading

What type of PR is this?

/kind bug
/kind flake

What this PR does / why we need it:
This builds on #96798 by reducing the expected throughput, since the main problem seems to be that the expected throughput is sometimes not achieved.

Which issue(s) this PR fixes:

Fixes #96710

Special notes for your reviewer:
As noted in my comments on #96798, I think the tests here need deep revision in order to become good tests; this PR is a tactical fix to stop the flakes.

Does this PR introduce a user-facing change?:

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

@k8s-ci-robot k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. kind/bug Categorizes issue or PR as related to a bug. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. kind/flake Categorizes issue or PR as related to a flaky test. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Nov 25, 2020
@k8s-ci-robot k8s-ci-robot added area/test sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. sig/testing Categorizes an issue or PR as relevant to SIG Testing. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Nov 25, 2020
@MikeSpreitzer MikeSpreitzer changed the title Flaky/apnf e2e drown test Tweak up flaking end-to-end tests of API Priority and Fairness Nov 25, 2020
@MikeSpreitzer
Copy link
Member Author

/cc @yue9944882
/cc @adtac
/cc @lavalamp
/cc @deads2k

@MikeSpreitzer
Copy link
Member Author

/uncc @smarterclayton

@k8s-ci-robot k8s-ci-robot removed the request for review from smarterclayton November 25, 2020 22:20
@MikeSpreitzer
Copy link
Member Author

/retest

@adtac
Copy link
Member

adtac commented Nov 26, 2020

your four commits LGTM, do you want to rebase on top of Min's latest commit? (I think Min has incorporated your review, so one of your commits might be redundant when you rebase)

@MikeSpreitzer MikeSpreitzer force-pushed the flaky/apnf-e2e-drown-test branch from f045377 to e7b0f58 Compare November 30, 2020 04:29
@MikeSpreitzer
Copy link
Member Author

The force-push to e7b0f58 is a rebase onto #96798

@MikeSpreitzer MikeSpreitzer mentioned this pull request Nov 30, 2020
Merged
@MikeSpreitzer
Copy link
Member Author

/retest

@adtac
Copy link
Member

adtac commented Nov 30, 2020

/lgtm
/retest
(retest to see if we can induce a flake)

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 30, 2020
@MikeSpreitzer
Copy link
Member Author

BTW, /retest only re-runs tests whose last run failed.

yue9944882
yue9944882 reviewed Dec 1, 2020
View reviewed changes
Copy link
Member

@yue9944882 yue9944882 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@fedebongio
Copy link
Contributor

/triage accepted

@k8s-ci-robot k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Dec 1, 2020
lavalamp
lavalamp reviewed Dec 4, 2020
View reviewed changes
test/e2e/apimachinery/flowcontrol.go
{username: "highqps", qps: 100.0, concurrencyMultiplier: 2.0},
{username: "lowqps", qps: 5.0, concurrencyMultiplier: 0.5},
{username: highQPSClientName, qps: 90, concurrencyMultiplier: 2.0, expectedCompletedPercentage: 0.90},
{username: lowQPSClientName, qps: 4, concurrencyMultiplier: 0.5, expectedCompletedPercentage: 0.90},
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't this .9 an increase over #96798's .75?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A recent failure:

Dec 4 15:38:28.824: client "highqps-apf-8126": got 36.6% completed requests, want at least 75.0%

So I think we'll need a followup.

Perhaps we need to use quite large share amounts to throttle the rest of the cluster.

@lavalamp
Copy link
Member

lavalamp commented Dec 4, 2020

/milestone v1.20

@k8s-ci-robot k8s-ci-robot added this to the v1.20 milestone Dec 4, 2020
@lavalamp
Copy link
Member

lavalamp commented Dec 4, 2020

/approve

Multiple reviewers seem to think this will help so let's merge it--I don't quite understand but I don't want to block this.

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: lavalamp, MikeSpreitzer

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 4, 2020
@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@k8s-ci-robot k8s-ci-robot merged commit e1c617a into kubernetes:master Dec 4, 2020
@bai
Copy link

bai commented Dec 7, 2020

/milestone v1.20

Hey there, just wanted to mention that this was after test freeze and the branch cut. If you’d like this in 1.20, it needs to be cherry picked and the release is tomorrow so this probably won’t make it until 1.20.1.

@adtac
Copy link
Member

adtac commented Dec 7, 2020

The tests have been disabled in 1.20 (#97075), so I don't think it matters too much whether or not we cherry-pick this.

@MikeSpreitzer MikeSpreitzer deleted the flaky/apnf-e2e-drown-test branch December 7, 2020 15:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yue9944882 yue9944882 yue9944882 left review comments

@lavalamp lavalamp lavalamp left review comments

@sttts sttts Awaiting requested review from sttts

@adtac adtac Awaiting requested review from adtac

@deads2k deads2k Awaiting requested review from deads2k

Assignees

@adtac adtac

@yue9944882 yue9944882

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/test cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. kind/flake Categorizes issue or PR as related to a flaky test. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. release-note-none Denotes a PR that doesn't merit a release note. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. sig/testing Categorizes an issue or PR as relevant to SIG Testing. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
v1.20
Development

Successfully merging this pull request may close these issues.

"highqps" not reaching 95% completion threshold on API priority and fairness ... , blocking gce-master-default
8 participants
@MikeSpreitzer @adtac @fedebongio @lavalamp @k8s-ci-robot @fejta-bot @bai @yue9944882