Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HOSTEDCP-2206: Improve Unit Test Runtime #5257

Merged
merged 3 commits into from
Dec 10, 2024
Merged

HOSTEDCP-2206: Improve Unit Test Runtime #5257

merged 3 commits into from
Dec 10, 2024

Conversation

bryan-cox
Copy link
Member

@bryan-cox bryan-cox commented Dec 8, 2024 •
edited
Loading

What this PR does / why we need it:
This PR:

  • reduces the unit test runtime by ~90% from my tests locally
  • reduces the CI runtime by ~72%*
  • fixes some minor codespelling mistakes picked up by pre-commit hooks

*calculated from 8 runs of this PR vs 8 successful runs of the unit tests without this PR

The majority of the time reduction is due from taking count from 25 to 1. Setting this value to 1 ensures all the unit tests are always run.

% go help testflag
...
	-count n
	    Run each test, benchmark, and fuzz seed n times (default 1).
	    If -cpu is set, run n times for each GOMAXPROCS value.
	    Examples are always run once. -count does not apply to
	    fuzz tests matched by -fuzz.

This PR also sets the parallel flag on unit tests according to the number of cores on the machine running the test. More info on the parallel flag can be found here:

% go help testflag
...
	-parallel n
	    Allow parallel execution of test functions that call t.Parallel, and
	    fuzz targets that call t.Parallel when running the seed corpus.
	    The value of this flag is the maximum number of tests to run
	    simultaneously.
	    While fuzzing, the value of this flag is the maximum number of
	    subprocesses that may call the fuzz function simultaneously, regardless of
	    whether T.Parallel is called.
	    By default, -parallel is set to the value of GOMAXPROCS.
	    Setting -parallel to values higher than GOMAXPROCS may cause degraded
	    performance due to CPU contention, especially when fuzzing.
	    Note that -parallel only applies within a single test binary.
	    The 'go test' command may run tests for different packages
	    in parallel as well, according to the setting of the -p flag
	    (see 'go help build').

Which issue(s) this PR fixes:
Fixes HOSTEDCP-2206

Checklist

  • Subject and description added to both, commit and PR.
  • Relevant issues have been referenced.
  • This change includes docs.
  • This change includes unit tests.

@bryan-cox
Improve UT runtime by count and parallel flags
b908874 
This commit improves unit test runtime by reducing the count flag from
25 to 1. The commit also adds the parallel flag so unit tests with t
.Parallel() can run in parallel. The max number of threads is set to the
 number of cores on the machine running the unit test.

Signed-off-by: Bryan Cox <brcox@redhat.com>
@bryan-cox
Fix spelling mistakes for codespell and whitespace
bf63250 
Signed-off-by: Bryan Cox <brcox@redhat.com>
@openshift-ci openshift-ci bot added area/documentation Indicates the PR includes changes for documentation area/testing Indicates the PR includes changes for e2e testing labels Dec 8, 2024
@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented Dec 8, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bryan-cox

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added approved Indicates a PR has been approved by an approver from all required OWNERS files. and removed do-not-merge/needs-area labels Dec 8, 2024
@bryan-cox
Copy link
Member Author

/test unit

5 similar comments
@bryan-cox
Copy link
Member Author

/test unit

@bryan-cox
Copy link
Member Author

/test unit

@bryan-cox
Copy link
Member Author

/test unit

@bryan-cox
Copy link
Member Author

/test unit

@bryan-cox
Copy link
Member Author

/test unit

@bryan-cox
Ignore api.md from trailing whitespace hook
757f54e 
Signed-off-by: Bryan Cox <brcox@redhat.com>
@bryan-cox bryan-cox changed the title Improve ut time HOSTEDCP-2206: Improve Unit Test Runtime Dec 8, 2024
@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Dec 8, 2024
@openshift-ci-robot
Copy link

openshift-ci-robot commented Dec 8, 2024 •
edited by openshift-ci bot
Loading

@bryan-cox: This pull request references HOSTEDCP-2206 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.19.0" version, but no target version was set.

In response to this:

What this PR does / why we need it:
This PR improves the unit test runtime by ~90% from my tests locally. The majority of the time reduction is due from taking count from 25 to 1. Setting this value to 1 ensures all the unit tests are always run.

% go help testflag
...
  -count n
      Run each test, benchmark, and fuzz seed n times (default 1).
      If -cpu is set, run n times for each GOMAXPROCS value.
      Examples are always run once. -count does not apply to
      fuzz tests matched by -fuzz.

This PR also sets the parallel flag on unit tests according to the number of cores on the machine running the test. More info on the parallel flag can be found here:

% go help testflag
...
  -parallel n
      Allow parallel execution of test functions that call t.Parallel, and
      fuzz targets that call t.Parallel when running the seed corpus.
      The value of this flag is the maximum number of tests to run
      simultaneously.
      While fuzzing, the value of this flag is the maximum number of
      subprocesses that may call the fuzz function simultaneously, regardless of
      whether T.Parallel is called.
      By default, -parallel is set to the value of GOMAXPROCS.
      Setting -parallel to values higher than GOMAXPROCS may cause degraded
      performance due to CPU contention, especially when fuzzing.
      Note that -parallel only applies within a single test binary.
      The 'go test' command may run tests for different packages
      in parallel as well, according to the setting of the -p flag
      (see 'go help build').

Which issue(s) this PR fixes (optional, use fixes #<issue_number>(, fixes #<issue_number>, ...) format, where issue_number might be a GitHub issue, or a Jira story:
Fixes #

Checklist

  • Subject and description added to both, commit and PR.
  • Relevant issues have been referenced.
  • This change includes docs.
  • This change includes unit tests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link

openshift-ci-robot commented Dec 8, 2024 •
edited by openshift-ci bot
Loading

@bryan-cox: This pull request references HOSTEDCP-2206 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.19.0" version, but no target version was set.

In response to this:

What this PR does / why we need it:
This PR improves the unit test runtime by ~90% from my tests locally. The majority of the time reduction is due from taking count from 25 to 1. Setting this value to 1 ensures all the unit tests are always run.

% go help testflag
...
  -count n
      Run each test, benchmark, and fuzz seed n times (default 1).
      If -cpu is set, run n times for each GOMAXPROCS value.
      Examples are always run once. -count does not apply to
      fuzz tests matched by -fuzz.

This PR also sets the parallel flag on unit tests according to the number of cores on the machine running the test. More info on the parallel flag can be found here:

% go help testflag
...
  -parallel n
      Allow parallel execution of test functions that call t.Parallel, and
      fuzz targets that call t.Parallel when running the seed corpus.
      The value of this flag is the maximum number of tests to run
      simultaneously.
      While fuzzing, the value of this flag is the maximum number of
      subprocesses that may call the fuzz function simultaneously, regardless of
      whether T.Parallel is called.
      By default, -parallel is set to the value of GOMAXPROCS.
      Setting -parallel to values higher than GOMAXPROCS may cause degraded
      performance due to CPU contention, especially when fuzzing.
      Note that -parallel only applies within a single test binary.
      The 'go test' command may run tests for different packages
      in parallel as well, according to the setting of the -p flag
      (see 'go help build').

Which issue(s) this PR fixes (optional, use fixes #<issue_number>(, fixes #<issue_number>, ...) format, where issue_number might be a GitHub issue, or a Jira story:
Fixes HOSTEDCP-2206

Checklist

  • Subject and description added to both, commit and PR.
  • Relevant issues have been referenced.
  • This change includes docs.
  • This change includes unit tests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link

openshift-ci-robot commented Dec 8, 2024 •
edited by openshift-ci bot
Loading

@bryan-cox: This pull request references HOSTEDCP-2206 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.19.0" version, but no target version was set.

In response to this:

What this PR does / why we need it:
This PR improves the unit test runtime by ~90% from my tests locally. The majority of the time reduction is due from taking count from 25 to 1. Setting this value to 1 ensures all the unit tests are always run.

% go help testflag
...
  -count n
      Run each test, benchmark, and fuzz seed n times (default 1).
      If -cpu is set, run n times for each GOMAXPROCS value.
      Examples are always run once. -count does not apply to
      fuzz tests matched by -fuzz.

This PR also sets the parallel flag on unit tests according to the number of cores on the machine running the test. More info on the parallel flag can be found here:

% go help testflag
...
  -parallel n
      Allow parallel execution of test functions that call t.Parallel, and
      fuzz targets that call t.Parallel when running the seed corpus.
      The value of this flag is the maximum number of tests to run
      simultaneously.
      While fuzzing, the value of this flag is the maximum number of
      subprocesses that may call the fuzz function simultaneously, regardless of
      whether T.Parallel is called.
      By default, -parallel is set to the value of GOMAXPROCS.
      Setting -parallel to values higher than GOMAXPROCS may cause degraded
      performance due to CPU contention, especially when fuzzing.
      Note that -parallel only applies within a single test binary.
      The 'go test' command may run tests for different packages
      in parallel as well, according to the setting of the -p flag
      (see 'go help build').

Which issue(s) this PR fixes:
Fixes HOSTEDCP-2206

Checklist

  • Subject and description added to both, commit and PR.
  • Relevant issues have been referenced.
  • This change includes docs.
  • This change includes unit tests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link

openshift-ci-robot commented Dec 8, 2024 •
edited by openshift-ci bot
Loading

@bryan-cox: This pull request references HOSTEDCP-2206 which is a valid jira issue.

In response to this:

What this PR does / why we need it:
This PR:

  • improves the unit test runtime by ~90% from my tests locally
  • improves the CI runtime by
  • fixes some minor codespelling mistakes picked up by pre-commit hooks

The majority of the time reduction is due from taking count from 25 to 1. Setting this value to 1 ensures all the unit tests are always run.

% go help testflag
...
  -count n
      Run each test, benchmark, and fuzz seed n times (default 1).
      If -cpu is set, run n times for each GOMAXPROCS value.
      Examples are always run once. -count does not apply to
      fuzz tests matched by -fuzz.

This PR also sets the parallel flag on unit tests according to the number of cores on the machine running the test. More info on the parallel flag can be found here:

% go help testflag
...
  -parallel n
      Allow parallel execution of test functions that call t.Parallel, and
      fuzz targets that call t.Parallel when running the seed corpus.
      The value of this flag is the maximum number of tests to run
      simultaneously.
      While fuzzing, the value of this flag is the maximum number of
      subprocesses that may call the fuzz function simultaneously, regardless of
      whether T.Parallel is called.
      By default, -parallel is set to the value of GOMAXPROCS.
      Setting -parallel to values higher than GOMAXPROCS may cause degraded
      performance due to CPU contention, especially when fuzzing.
      Note that -parallel only applies within a single test binary.
      The 'go test' command may run tests for different packages
      in parallel as well, according to the setting of the -p flag
      (see 'go help build').

Which issue(s) this PR fixes:
Fixes HOSTEDCP-2206

Checklist

  • Subject and description added to both, commit and PR.
  • Relevant issues have been referenced.
  • This change includes docs.
  • This change includes unit tests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link

openshift-ci-robot commented Dec 8, 2024 •
edited by openshift-ci bot
Loading

@bryan-cox: This pull request references HOSTEDCP-2206 which is a valid jira issue.

In response to this:

What this PR does / why we need it:
This PR:

  • improves the unit test runtime by ~90% from my tests locally
  • improves the CI runtime by ~72%*
  • fixes some minor codespelling mistakes picked up by pre-commit hooks
  • calculated from 8 runs of this PR vs 8 successful runs of the unit tests without this PR

The majority of the time reduction is due from taking count from 25 to 1. Setting this value to 1 ensures all the unit tests are always run.

% go help testflag
...
  -count n
      Run each test, benchmark, and fuzz seed n times (default 1).
      If -cpu is set, run n times for each GOMAXPROCS value.
      Examples are always run once. -count does not apply to
      fuzz tests matched by -fuzz.

This PR also sets the parallel flag on unit tests according to the number of cores on the machine running the test. More info on the parallel flag can be found here:

% go help testflag
...
  -parallel n
      Allow parallel execution of test functions that call t.Parallel, and
      fuzz targets that call t.Parallel when running the seed corpus.
      The value of this flag is the maximum number of tests to run
      simultaneously.
      While fuzzing, the value of this flag is the maximum number of
      subprocesses that may call the fuzz function simultaneously, regardless of
      whether T.Parallel is called.
      By default, -parallel is set to the value of GOMAXPROCS.
      Setting -parallel to values higher than GOMAXPROCS may cause degraded
      performance due to CPU contention, especially when fuzzing.
      Note that -parallel only applies within a single test binary.
      The 'go test' command may run tests for different packages
      in parallel as well, according to the setting of the -p flag
      (see 'go help build').

Which issue(s) this PR fixes:
Fixes HOSTEDCP-2206

Checklist

  • Subject and description added to both, commit and PR.
  • Relevant issues have been referenced.
  • This change includes docs.
  • This change includes unit tests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link

openshift-ci-robot commented Dec 8, 2024 •
edited by openshift-ci bot
Loading

@bryan-cox: This pull request references HOSTEDCP-2206 which is a valid jira issue.

In response to this:

What this PR does / why we need it:
This PR:

  • improves the unit test runtime by ~90% from my tests locally
  • improves the CI runtime by ~72%*
  • fixes some minor codespelling mistakes picked up by pre-commit hooks

*calculated from 8 runs of this PR vs 8 successful runs of the unit tests without this PR

The majority of the time reduction is due from taking count from 25 to 1. Setting this value to 1 ensures all the unit tests are always run.

% go help testflag
...
  -count n
      Run each test, benchmark, and fuzz seed n times (default 1).
      If -cpu is set, run n times for each GOMAXPROCS value.
      Examples are always run once. -count does not apply to
      fuzz tests matched by -fuzz.

This PR also sets the parallel flag on unit tests according to the number of cores on the machine running the test. More info on the parallel flag can be found here:

% go help testflag
...
  -parallel n
      Allow parallel execution of test functions that call t.Parallel, and
      fuzz targets that call t.Parallel when running the seed corpus.
      The value of this flag is the maximum number of tests to run
      simultaneously.
      While fuzzing, the value of this flag is the maximum number of
      subprocesses that may call the fuzz function simultaneously, regardless of
      whether T.Parallel is called.
      By default, -parallel is set to the value of GOMAXPROCS.
      Setting -parallel to values higher than GOMAXPROCS may cause degraded
      performance due to CPU contention, especially when fuzzing.
      Note that -parallel only applies within a single test binary.
      The 'go test' command may run tests for different packages
      in parallel as well, according to the setting of the -p flag
      (see 'go help build').

Which issue(s) this PR fixes:
Fixes HOSTEDCP-2206

Checklist

  • Subject and description added to both, commit and PR.
  • Relevant issues have been referenced.
  • This change includes docs.
  • This change includes unit tests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@bryan-cox bryan-cox mentioned this pull request Dec 8, 2024
Merged
4 tasks
@openshift-ci-robot
Copy link

openshift-ci-robot commented Dec 9, 2024 •
edited by openshift-ci bot
Loading

@bryan-cox: This pull request references HOSTEDCP-2206 which is a valid jira issue.

In response to this:

What this PR does / why we need it:
This PR:

  • reduces the unit test runtime by ~90% from my tests locally
  • reduces the CI runtime by ~72%*
  • fixes some minor codespelling mistakes picked up by pre-commit hooks

*calculated from 8 runs of this PR vs 8 successful runs of the unit tests without this PR

The majority of the time reduction is due from taking count from 25 to 1. Setting this value to 1 ensures all the unit tests are always run.

% go help testflag
...
  -count n
      Run each test, benchmark, and fuzz seed n times (default 1).
      If -cpu is set, run n times for each GOMAXPROCS value.
      Examples are always run once. -count does not apply to
      fuzz tests matched by -fuzz.

This PR also sets the parallel flag on unit tests according to the number of cores on the machine running the test. More info on the parallel flag can be found here:

% go help testflag
...
  -parallel n
      Allow parallel execution of test functions that call t.Parallel, and
      fuzz targets that call t.Parallel when running the seed corpus.
      The value of this flag is the maximum number of tests to run
      simultaneously.
      While fuzzing, the value of this flag is the maximum number of
      subprocesses that may call the fuzz function simultaneously, regardless of
      whether T.Parallel is called.
      By default, -parallel is set to the value of GOMAXPROCS.
      Setting -parallel to values higher than GOMAXPROCS may cause degraded
      performance due to CPU contention, especially when fuzzing.
      Note that -parallel only applies within a single test binary.
      The 'go test' command may run tests for different packages
      in parallel as well, according to the setting of the -p flag
      (see 'go help build').

Which issue(s) this PR fixes:
Fixes HOSTEDCP-2206

Checklist

  • Subject and description added to both, commit and PR.
  • Relevant issues have been referenced.
  • This change includes docs.
  • This change includes unit tests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@sjenning
Copy link
Contributor

sjenning commented Dec 9, 2024

/lgtm

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Dec 9, 2024
@bryan-cox
Copy link
Member Author

Documenting this for future us 😄

We discussed leaving count at 1 for now during a grooming call today. If there is an issue with a unit test, that would have been caught with count being higher than one, we will parameterize the count value to 1 locally and change it to something like 10 in the unit tests.

@bryan-cox
Copy link
Member Author

/retest-required

@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented Dec 9, 2024 •
edited
Loading

@bryan-cox: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/okd-scos-e2e-aws-ovn 757f54e link false /test okd-scos-e2e-aws-ovn

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@bryan-cox
Copy link
Member Author

/test e2e-aws-4-18

@openshift-merge-bot openshift-merge-bot bot merged commit 76ef394 into openshift:main Dec 10, 2024
12 of 13 checks passed
@bryan-cox bryan-cox deleted the improve-ut-time branch December 10, 2024 02:23
@openshift-bot
Copy link

[ART PR BUILD NOTIFIER]

Distgit: hypershift
This PR has been included in build ose-hypershift-container-v4.19.0-202412100538.p0.g76ef394.assembly.stream.el9.
All builds following this will include this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@isco-rodriguez isco-rodriguez Awaiting requested review from isco-rodriguez

@sjenning sjenning Awaiting requested review from sjenning

Assignees

@sjenning sjenning

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/documentation Indicates the PR includes changes for documentation area/testing Indicates the PR includes changes for e2e testing jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@bryan-cox @openshift-ci-robot @sjenning @openshift-bot