Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement the integration tests for requeueing scenarios #122305

Closed
10 of 12 tasks
Tracked by #122597
sanposhiho opened this issue Dec 14, 2023 · 77 comments · Fixed by #127139 or #127354
Closed
10 of 12 tasks
Tracked by #122597

Implement the integration tests for requeueing scenarios #122305

sanposhiho opened this issue Dec 14, 2023 · 77 comments · Fixed by #127139 or #127354
Labels
kind/feature Categorizes issue or PR as related to a new feature. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling.

Comments

@sanposhiho
Copy link
Member

sanposhiho commented Dec 14, 2023

/sig scheduling
/priority important-soon
/kind feature

Part of: #122284 (comment).

We have few tests to check the scheduler's requeueing scenario in the integration test.
We should have the one so that we can catch a bug like #122284, which only happens the scheduling queue is actually working.


Here's the list of plugins with QueueingHint.
Basically, I believe we can just pile up the test cases in the existing TestCoreResourceEnqueue.
Also, note that we have to cover major scenarios only, don't have to cover minor ones within the integration test.

Please comment in this issue before starting to work on so that we can avoid many people from working on the same thing!

@k8s-ci-robot k8s-ci-robot added sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. kind/feature Categorizes issue or PR as related to a new feature. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Dec 14, 2023
@k8s-ci-robot
Copy link
Contributor

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@sanposhiho
Copy link
Member Author

/assign @carlory @sanposhiho

@sanposhiho
Copy link
Member Author

sanposhiho commented Dec 16, 2023

@carlory

A basic thought that I have now is that we have to cover the most common scenario only in an integration test, we don't have to cover all scenarios per each plugin.
So, for example, for nodeaffinity, we can just have two scenarios in which Pod is supposed to be requeued to activeQ/backoffQ (= node satisfying nodeaffinity is created), vs is supposed not to be requeued (= node unrelated to nodeaffinity is created). No need to cover other minor scenarios (e.g., a scenario addedNodeSelector comes into play, etc)

Plus, it'd be great if we could have a scenario where preCheck scenario is covered.
For example:

  1. Pod with nodeaffinity is created
  2. (1) is rejected because no node satisfies nodeaffinity
  3. we create a new node that satisfies nodeaffinity, but has taint.
  4. nodeadd event from (3) should be filtered out by precheck.
  5. we remove taint from node.
  6. nodeupdate event from (5) requeue the pod.

@carlory
Copy link
Member

carlory commented Dec 16, 2023

For example:
...

@sanposhiho There's already an existing intergation test. it follows similar steps.

https://github.com/carlory/kubernetes/blob/d36a7089cd9fa2abedd62f0964a42c21f837c53a/test/integration/scheduler/scheduler_test.go#L529

@sanposhiho
Copy link
Member Author

sanposhiho commented Dec 16, 2023

Then, we may want to refactor that test to be a more general one.
What I want to make sure (eventually) is that all plugins take preCheck into consideration in EventsToRegisrter (+ QHint). That TestNodeEvents makes sure only noderesourcefit currently.

@sanposhiho
Copy link
Member Author

As part of #122292, I implemented the e2e test, which would be a good starting point of this.

@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle stale
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 6, 2024
@sanposhiho
Copy link
Member Author

/remove-lifecycle stale

#122292 is implemented, we have to cultivate the test cases there for existing QHints.

@k8s-ci-robot k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 7, 2024
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle stale
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jul 6, 2024
@sanposhiho
Copy link
Member Author

/remove-lifecycle stale

Still important

@k8s-ci-robot k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jul 6, 2024
@Gekko0114
Copy link
Member

Hi @sanposhiho

I can work it. Could you assign any item to me?

@sanposhiho
Copy link
Member Author

Yes, name the ones you want.

@bells17
Copy link
Contributor

bells17 commented Sep 1, 2024

Hi @sanposhiho
May I work on adding tests for VolumeBinding?

@Gekko0114
Copy link
Member

Can you assign volumezone to me?

@sanposhiho
Copy link
Member Author

/reopen

mistake again!!!! 😓

@k8s-ci-robot
Copy link
Contributor

@sanposhiho: Reopened this issue.

In response to this:

/reopen

mistake again!!!! 😓

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot reopened this Sep 19, 2024
@sanposhiho
Copy link
Member Author

@mipplo @bells17 @utam0k Friendly reminder for unsubmitted PRs.
We have 1 month left until the deadline though, let us know if you may not be able to make it so that someone can take over.

@sanposhiho
Copy link
Member Author

@mipplo Are you still planning to work on it? I guess you deleted your comment from the thread here.

@googs1025
Copy link
Member

/cc

@utam0k
Copy link
Member

utam0k commented Oct 5, 2024

@mipplo @bells17 @utam0k Friendly reminder for unsubmitted PRs. We have 1 month left until the deadline though, let us know if you may not be able to make it so that someone can take over.

Sorry for the late. I'm working on mine.

@saku3
Copy link
Contributor

saku3 commented Oct 5, 2024

@bells17 Are you still working on volumebinding? I can take it over if you're busy.

@sanposhiho
Copy link
Member Author

@saku3 You can take it over. Since the volume binding is one of the biggest plugins, we should start asap.

@sanposhiho
Copy link
Member Author

Also, is anyone here interested in taking over the taint toleration plugin?

@bells17
Copy link
Contributor

bells17 commented Oct 7, 2024

@saku3 cc: @sanposhiho I'm sorry, but it seems I won't have much time available, so it would be very helpful if you could do it for me.

@bells17
Copy link
Contributor

bells17 commented Oct 7, 2024

/unassign @bells17

@unvavo
Copy link
Contributor

unvavo commented Oct 8, 2024

@sanposhiho

Also, is anyone here interested in taking over the taint toleration plugin?

I am interested in that. Can I take it over?

@sanposhiho
Copy link
Member Author

Yes.

@googs1025
Copy link
Member

googs1025 commented Oct 8, 2024

Can we scope out DRA from this issue? meaning, do we already have some integration tests for requeueing for DRA?

I am using scheduler_perf to write integration tests for DRA. It tends to be very good at finding unexpected race conditions or missing code (like the queuing hint for pod changes).

With unit tests and "traditional" integration tests one can verify that certain known corner cases get handled, i.e. one can write regression tests.

@sanposhiho @pohly
I know Patrick is currently implementing dra related integration tests in scheduler_perf.
Do we still need integration tests for dra's requeue scenario?
I can take this if we still need to.

@pohly
Copy link
Contributor

pohly commented Oct 17, 2024

I don't think that such specific integration tests are needed for DRA, but I do think that having them won't hurt 😄 So yes, if you have time and want to help, then please do.

@sanposhiho
Copy link
Member Author

@utam0k Do you need any help with two that you have?

@utam0k
Copy link
Member

utam0k commented Oct 28, 2024

@utam0k Do you need any help with two that you have?

VoulumeRestriction has been completed. And the other one is working on it. I'll send out my fix this week.
#127892

@sanposhiho
Copy link
Member Author

sanposhiho commented Oct 28, 2024

Missed the volume restriction is done already in the list!
Perfect, thanks.

@sanposhiho
Copy link
Member Author

@utam0k @saku3 Do you still think you'll be able to merge the PRs until the code freeze, tge end of this week?

@utam0k
Copy link
Member

utam0k commented Nov 4, 2024

@utam0k @saku3 Do you still think you'll be able to merge the PRs until the code freeze, tge end of this week?

I've almost done it, but I don't know how to do a few things. Can I get some advice? I'll push my code for now on the PR.

@sanposhiho
Copy link
Member Author

/close

All done! Thanks everyone!

@k8s-ci-robot
Copy link
Contributor

@sanposhiho: Closing this issue.

In response to this:

/close

All done! Thanks everyone!

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/feature Categorizes issue or PR as related to a new feature. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling.
Projects
None yet