Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: reduced delays and syncs in functional tests to run faster #6268

Closed
wants to merge 7 commits into from

Conversation

knst
Copy link
Collaborator

@knst knst commented Sep 12, 2024

Issue being fixed or feature implemented

Functional tests take too long time to run.

What was done?

  • increased robustness feature_llmq_simplepose.py by adding missing bump for mocktime during block generations
  • removed sleep(1) from each stage of mine_quorum
  • removed sleep(6) from final stage of mine_cycled_quorum
  • size of batch for block generation in feature_asset_locks.py and in activate_fork_by_name() increased from 10 blocks to 50 blocks
  • moved governance's functional tests to "60 seconds+" category because they always the last one to wait if running more than 10 jobs at once
    Plus extra refactoring which removes dead and commented code from test_framework.py

How Has This Been Tested?

Locally, the functional tests speed up with these fixes for 15% for overall time and 20% for accumulated time
test/functional/test_runner.py -j20

Before:

ALL                                                | ✓ Passed  | 7860 s (accumulated)  
Runtime: 481 s

After:

ALL                                                | ✓ Passed | 6237 s (accumulated) 
Runtime: 416 s

CI tsan job speeds up for 5 minutes in absolute time (~5%) and 1000 seconds in accumulated time.

ALL                                                | ✓ Passed  | 23854 s (accumulated) 
Runtime: 6249 s

old version vs new version

ALL                                                | ✓ Passed  | 22901 s (accumulated) 
Runtime: 5962 s

Breaking Changes

N/A

Checklist:

  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have added or updated relevant unit/integration/functional/e2e tests
  • I have made corresponding changes to the documentation
  • I have assigned this pull request to a milestone

@knst knst added this to the 21.2 milestone Sep 12, 2024
@knst knst force-pushed the functional-tests-delays-p2 branch from 8b44a6d to cea2e3f Compare September 12, 2024 12:32
@knst knst marked this pull request as draft September 12, 2024 12:57
@knst knst force-pushed the functional-tests-delays-p2 branch from cea2e3f to b8fc9cc Compare September 13, 2024 05:59
@knst knst changed the title feat(perf): reduced delays and syncs in functional tests to run faster perf: reduced delays and syncs in functional tests to run faster Sep 13, 2024
@knst knst force-pushed the functional-tests-delays-p2 branch from b8fc9cc to 595900c Compare September 13, 2024 06:55
…in functional tests

It significantly improve speed of forks activation because reduces overhead for block generations
Bigger batch size can cause time-outs for RPC for tsan job (time-out is 30 seconds)
@knst knst force-pushed the functional-tests-delays-p2 branch from 356c142 to e382dfc Compare September 14, 2024 17:03
@knst knst force-pushed the functional-tests-delays-p2 branch from e382dfc to 4f15417 Compare September 14, 2024 17:06
@knst knst marked this pull request as ready for review September 14, 2024 17:35
Copy link

@UdjinM6 UdjinM6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

feature_llmq_simplepose.py starts to fail much more often for me locally after d561419 (when running 8 of them in parallel). 5d4301d f72a70e fixes that.

Copy link
Member

@PastaPastaPasta PastaPastaPasta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re-request with udjin applied

@UdjinM6
Copy link

UdjinM6 commented Sep 17, 2024

Looks like we are getting different results here, my commit makes things a bit less stable for knst for some reason. @PastaPastaPasta pls confirm the issue (and that my commit actually helps).

@knst
Copy link
Collaborator Author

knst commented Sep 18, 2024

Looks like we are getting different results here, my commit makes things a bit less stable for knst for some reason.

on laptop udjin's patch continue to fail, but on PC (different kernel version, different hardware, same compiler) it doesn't fail that much.

Let's see how CI behaves

@knst knst closed this Sep 18, 2024
@knst knst deleted the functional-tests-delays-p2 branch September 18, 2024 18:14
PastaPastaPasta added a commit that referenced this pull request Sep 25, 2024
…n faster

874ef8c fix: mine_quorum_no_checks -> mine_quorum_less_checks: do some checks to make sure quorums are mined correctly (UdjinM6)
4f636f4 fix: re-order functional tests: move governance to 60+seconds category (Konstantin Akimov)
fe49f3f refactor: removed dead and commented code from test_framework.py (Konstantin Akimov)
cd1958c perf: removed sleep(6) from mine_cycle_quorum in functional tests (Konstantin Akimov)
3f17a01 fix: bump mocktime in simplepose when generating blocks to improve robustness (Konstantin Akimov)
132d95e perf: remove sleep(1) from each step of quorum creation in functional tests (Konstantin Akimov)
4c57ad1 chore: increase batch size from 10 to 50 for faster block generation in functional tests (Konstantin Akimov)

Pull request description:

  ## Issue being fixed or feature implemented
  Functional tests take too long time to run.

  (PR is recreated from #6268 because CI is broken)

  ## What was done?
   - increased robustness `feature_llmq_simplepose.py` by adding missing bump for mocktime during block generations
   - removed sleep(1) from each stage of mine_quorum
   - removed sleep(6) from final stage of mine_cycled_quorum
   - size of batch for block generation in `feature_asset_locks.py` and in `activate_fork_by_name()` increased from 10 blocks to 50 blocks
   - moved governance's functional tests to "60 seconds+" category because they always the last one to wait if running more than 10 jobs at once
  Plus extra refactoring which removes dead and commented code from test_framework.py

  ## How Has This Been Tested?
  Locally, the functional tests speed up with these fixes for 15% for overall time and 20% for accumulated time
  `test/functional/test_runner.py -j20`

  Before:
  ```
  ALL                                                | ✓ Passed  | 7860 s (accumulated)
  Runtime: 481 s
  ```

  After:
  ```
  ALL                                                | ✓ Passed | 6237 s (accumulated)
  Runtime: 416 s
  ```

  ---
  CI tsan job speeds up for 5 minutes in absolute time (~5%) and 1000 seconds in accumulated time.
  ```
  ALL                                                | ✓ Passed  | 23854 s (accumulated)
  Runtime: 6249 s
  ```
  ↑ [old version](https://gitlab.com/dashpay/dash/-/jobs/7822664869) vs [new version](https://gitlab.com/dashpay/dash/-/jobs/7825461091) ↓
  ```
  ALL                                                | ✓ Passed  | 22901 s (accumulated)
  Runtime: 5962 s
  ```

  ## Breaking Changes
  N/A

  ## Checklist:
  - [x] I have performed a self-review of my own code
  - [x] I have commented my code, particularly in hard-to-understand areas
  - [x] I have added or updated relevant unit/integration/functional/e2e tests
  - [x] I have made corresponding changes to the documentation
  - [x] I have assigned this pull request to a milestone

ACKs for top commit:
  UdjinM6:
    utACK 874ef8c
  PastaPastaPasta:
    utACK 874ef8c

Tree-SHA512: 514fa2fb32abd59c90f63b68fccc8c3d3b6d16b0b6ad7459c4a348825815e7d3012177565dea1f70b8a1f28ede1a297f91361365454d1be85955e77260451cf5
@UdjinM6 UdjinM6 modified the milestones: 21.2, 22 Oct 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants