Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BugFix] Refactor reductions #1968

Merged
merged 5 commits into from
Feb 27, 2024
Merged

[BugFix] Refactor reductions #1968

merged 5 commits into from
Feb 27, 2024

Conversation

vmoens
Copy link
Contributor

@vmoens vmoens commented Feb 27, 2024 •
edited
Loading

cc @albertbou92

The tests are failing because when collecting loss.action_loss(...) the tensor is not reduced.

I think if the sub-loss is a public method the reduction should happen there.

Also we should only reduce the losses, not the metadata.

The PR solves both these issues

@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Feb 27, 2024 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/1968

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 2 Unrelated Failures

As of commit 00fffab with merge base b8ad113 (image):

NEW FAILURE - The following job has failed:

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Feb 27, 2024
@vmoens vmoens changed the title [Feature] Use non-default mp start method in ParallelEnv [[BugFix] Refactor reductions Feb 27, 2024
@vmoens vmoens added the bug Something isn't working label Feb 27, 2024
@vmoens vmoens changed the title [[BugFix] Refactor reductions [BugFix] Refactor reductions Feb 27, 2024
@vmoens vmoens marked this pull request as ready for review February 27, 2024 00:50
@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 27, 2024 •
edited
Loading

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of CPU Benchmark Tests

Total Benchmarks: 89. Improved: $\large\color{#35bf28}5$. Worsened: $\large\color{#d91a1a}2$.

Expand to view detailed results
Name Max Mean Ops Ops on Repo HEAD Change
test_single 61.3571ms 60.9329ms 16.4115 Ops/s 16.4994 Ops/s $\color{#d91a1a}-0.53\%$
test_sync 34.5857ms 32.9518ms 30.3474 Ops/s 29.4725 Ops/s $\color{#35bf28}+2.97\%$
test_async 60.7826ms 29.9556ms 33.3827 Ops/s 31.6680 Ops/s $\textbf{\color{#35bf28}+5.41\%}$
test_simple 0.4847s 0.4336s 2.3063 Ops/s 2.3167 Ops/s $\color{#d91a1a}-0.45\%$
test_transformed 0.6246s 0.5781s 1.7297 Ops/s 1.7429 Ops/s $\color{#d91a1a}-0.76\%$
test_serial 1.4471s 1.4028s 0.7128 Ops/s 0.7085 Ops/s $\color{#35bf28}+0.61\%$
test_parallel 1.4347s 1.3894s 0.7197 Ops/s 0.7142 Ops/s $\color{#35bf28}+0.77\%$
test_step_mdp_speed[True-True-True-True-True] 0.1391ms 21.6034μs 46.2891 KOps/s 47.3545 KOps/s $\color{#d91a1a}-2.25\%$
test_step_mdp_speed[True-True-True-True-False] 35.4670μs 13.2123μs 75.6870 KOps/s 77.1513 KOps/s $\color{#d91a1a}-1.90\%$
test_step_mdp_speed[True-True-True-False-True] 34.3740μs 12.6774μs 78.8808 KOps/s 80.7549 KOps/s $\color{#d91a1a}-2.32\%$
test_step_mdp_speed[True-True-True-False-False] 31.6100μs 7.7510μs 129.0150 KOps/s 133.2336 KOps/s $\color{#d91a1a}-3.17\%$
test_step_mdp_speed[True-True-False-True-True] 74.4490μs 23.2589μs 42.9942 KOps/s 44.9120 KOps/s $\color{#d91a1a}-4.27\%$
test_step_mdp_speed[True-True-False-True-False] 48.0500μs 14.6073μs 68.4589 KOps/s 71.2604 KOps/s $\color{#d91a1a}-3.93\%$
test_step_mdp_speed[True-True-False-False-True] 38.5220μs 14.0554μs 71.1470 KOps/s 74.4642 KOps/s $\color{#d91a1a}-4.45\%$
test_step_mdp_speed[True-True-False-False-False] 28.7630μs 9.0756μs 110.1861 KOps/s 114.5829 KOps/s $\color{#d91a1a}-3.84\%$
test_step_mdp_speed[True-False-True-True-True] 57.5080μs 24.4921μs 40.8295 KOps/s 42.1819 KOps/s $\color{#d91a1a}-3.21\%$
test_step_mdp_speed[True-False-True-True-False] 59.8920μs 15.8802μs 62.9714 KOps/s 64.5789 KOps/s $\color{#d91a1a}-2.49\%$
test_step_mdp_speed[True-False-True-False-True] 35.9970μs 13.9238μs 71.8195 KOps/s 73.7901 KOps/s $\color{#d91a1a}-2.67\%$
test_step_mdp_speed[True-False-True-False-False] 29.1150μs 9.0257μs 110.7944 KOps/s 114.6165 KOps/s $\color{#d91a1a}-3.33\%$
test_step_mdp_speed[True-False-False-True-True] 65.7630μs 25.4941μs 39.2247 KOps/s 40.1711 KOps/s $\color{#d91a1a}-2.36\%$
test_step_mdp_speed[True-False-False-True-False] 48.6510μs 17.1724μs 58.2330 KOps/s 60.4765 KOps/s $\color{#d91a1a}-3.71\%$
test_step_mdp_speed[True-False-False-False-True] 36.0670μs 14.9552μs 66.8662 KOps/s 67.9402 KOps/s $\color{#d91a1a}-1.58\%$
test_step_mdp_speed[True-False-False-False-False] 35.7970μs 10.1945μs 98.0924 KOps/s 101.8102 KOps/s $\color{#d91a1a}-3.65\%$
test_step_mdp_speed[False-True-True-True-True] 54.3820μs 24.1704μs 41.3729 KOps/s 42.3894 KOps/s $\color{#d91a1a}-2.40\%$
test_step_mdp_speed[False-True-True-True-False] 39.8950μs 15.8477μs 63.1006 KOps/s 65.0700 KOps/s $\color{#d91a1a}-3.03\%$
test_step_mdp_speed[False-True-True-False-True] 55.1430μs 15.9997μs 62.5013 KOps/s 62.9643 KOps/s $\color{#d91a1a}-0.74\%$
test_step_mdp_speed[False-True-True-False-False] 32.8320μs 10.2493μs 97.5677 KOps/s 100.6280 KOps/s $\color{#d91a1a}-3.04\%$
test_step_mdp_speed[False-True-False-True-True] 36.0270μs 25.5931μs 39.0730 KOps/s 39.2061 KOps/s $\color{#d91a1a}-0.34\%$
test_step_mdp_speed[False-True-False-True-False] 45.2650μs 17.1333μs 58.3657 KOps/s 59.8954 KOps/s $\color{#d91a1a}-2.55\%$
test_step_mdp_speed[False-True-False-False-True] 46.4270μs 17.3287μs 57.7077 KOps/s 58.6944 KOps/s $\color{#d91a1a}-1.68\%$
test_step_mdp_speed[False-True-False-False-False] 44.1130μs 11.3878μs 87.8135 KOps/s 90.2582 KOps/s $\color{#d91a1a}-2.71\%$
test_step_mdp_speed[False-False-True-True-True] 52.2480μs 26.8012μs 37.3118 KOps/s 38.3473 KOps/s $\color{#d91a1a}-2.70\%$
test_step_mdp_speed[False-False-True-True-False] 45.8060μs 18.2527μs 54.7865 KOps/s 55.8271 KOps/s $\color{#d91a1a}-1.86\%$
test_step_mdp_speed[False-False-True-False-True] 38.6420μs 17.2924μs 57.8288 KOps/s 58.6546 KOps/s $\color{#d91a1a}-1.41\%$
test_step_mdp_speed[False-False-True-False-False] 31.4490μs 11.3347μs 88.2244 KOps/s 89.5033 KOps/s $\color{#d91a1a}-1.43\%$
test_step_mdp_speed[False-False-False-True-True] 68.5780μs 27.6848μs 36.1209 KOps/s 36.9673 KOps/s $\color{#d91a1a}-2.29\%$
test_step_mdp_speed[False-False-False-True-False] 47.4580μs 19.5299μs 51.2035 KOps/s 52.7571 KOps/s $\color{#d91a1a}-2.94\%$
test_step_mdp_speed[False-False-False-False-True] 45.2350μs 18.2483μs 54.7997 KOps/s 55.9557 KOps/s $\color{#d91a1a}-2.07\%$
test_step_mdp_speed[False-False-False-False-False] 35.6270μs 12.5517μs 79.6703 KOps/s 82.3678 KOps/s $\color{#d91a1a}-3.27\%$
test_values[generalized_advantage_estimate-True-True] 12.3558ms 9.4353ms 105.9844 Ops/s 108.6433 Ops/s $\color{#d91a1a}-2.45\%$
test_values[vec_generalized_advantage_estimate-True-True] 36.6919ms 33.1999ms 30.1206 Ops/s 28.7498 Ops/s $\color{#35bf28}+4.77\%$
test_values[td0_return_estimate-False-False] 0.2258ms 0.1652ms 6.0537 KOps/s 6.0766 KOps/s $\color{#d91a1a}-0.38\%$
test_values[td1_return_estimate-False-False] 30.7700ms 23.5686ms 42.4293 Ops/s 43.3936 Ops/s $\color{#d91a1a}-2.22\%$
test_values[vec_td1_return_estimate-False-False] 48.3085ms 33.8465ms 29.5452 Ops/s 28.6066 Ops/s $\color{#35bf28}+3.28\%$
test_values[td_lambda_return_estimate-True-False] 37.0508ms 33.9022ms 29.4966 Ops/s 30.0673 Ops/s $\color{#d91a1a}-1.90\%$
test_values[vec_td_lambda_return_estimate-True-False] 34.7041ms 33.2610ms 30.0652 Ops/s 28.5768 Ops/s $\textbf{\color{#35bf28}+5.21\%}$
test_gae_speed[generalized_advantage_estimate-False-1-512] 10.3006ms 8.1775ms 122.2861 Ops/s 122.1459 Ops/s $\color{#35bf28}+0.11\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512] 2.1200ms 1.8792ms 532.1469 Ops/s 542.0489 Ops/s $\color{#d91a1a}-1.83\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512] 0.4262ms 0.3516ms 2.8440 KOps/s 2.9292 KOps/s $\color{#d91a1a}-2.91\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512] 41.3653ms 38.7967ms 25.7754 Ops/s 23.5150 Ops/s $\textbf{\color{#35bf28}+9.61\%}$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512] 3.7827ms 3.0327ms 329.7415 Ops/s 332.6822 Ops/s $\color{#d91a1a}-0.88\%$
test_dqn_speed 6.7430ms 1.3284ms 752.7951 Ops/s 740.0580 Ops/s $\color{#35bf28}+1.72\%$
test_ddpg_speed 3.0015ms 2.6417ms 378.5398 Ops/s 367.0727 Ops/s $\color{#35bf28}+3.12\%$
test_sac_speed 73.0230ms 8.6717ms 115.3177 Ops/s 123.7322 Ops/s $\textbf{\color{#d91a1a}-6.80\%}$
test_redq_speed 14.6003ms 12.9154ms 77.4267 Ops/s 76.5797 Ops/s $\color{#35bf28}+1.11\%$
test_redq_deprec_speed 13.6666ms 12.9078ms 77.4725 Ops/s 77.9847 Ops/s $\color{#d91a1a}-0.66\%$
test_td3_speed 8.5125ms 8.0276ms 124.5706 Ops/s 124.1387 Ops/s $\color{#35bf28}+0.35\%$
test_cql_speed 37.0032ms 35.7520ms 27.9705 Ops/s 27.8690 Ops/s $\color{#35bf28}+0.36\%$
test_a2c_speed 8.2958ms 7.2784ms 137.3934 Ops/s 136.3433 Ops/s $\color{#35bf28}+0.77\%$
test_ppo_speed 8.9053ms 7.5453ms 132.5321 Ops/s 128.1275 Ops/s $\color{#35bf28}+3.44\%$
test_reinforce_speed 7.4692ms 6.4880ms 154.1307 Ops/s 153.6782 Ops/s $\color{#35bf28}+0.29\%$
test_iql_speed 32.8662ms 32.1010ms 31.1517 Ops/s 30.7924 Ops/s $\color{#35bf28}+1.17\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 2.5921ms 2.1797ms 458.7726 Ops/s 468.8643 Ops/s $\color{#d91a1a}-2.15\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.8390ms 0.4901ms 2.0402 KOps/s 2.0299 KOps/s $\color{#35bf28}+0.51\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.8723ms 0.4683ms 2.1355 KOps/s 2.1544 KOps/s $\color{#d91a1a}-0.88\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 2.5829ms 2.1781ms 459.1257 Ops/s 469.1928 Ops/s $\color{#d91a1a}-2.15\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 0.9823ms 0.4842ms 2.0651 KOps/s 2.0727 KOps/s $\color{#d91a1a}-0.37\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.6155ms 0.4572ms 2.1873 KOps/s 2.1838 KOps/s $\color{#35bf28}+0.16\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 3.3469ms 2.2845ms 437.7265 Ops/s 440.6586 Ops/s $\color{#d91a1a}-0.67\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 1.2313ms 0.5981ms 1.6720 KOps/s 1.6609 KOps/s $\color{#35bf28}+0.67\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.8391ms 0.5783ms 1.7292 KOps/s 1.7339 KOps/s $\color{#d91a1a}-0.27\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 3.3028ms 2.1800ms 458.7177 Ops/s 463.1026 Ops/s $\color{#d91a1a}-0.95\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.7310ms 0.4903ms 2.0395 KOps/s 2.0561 KOps/s $\color{#d91a1a}-0.81\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.6239ms 0.4631ms 2.1594 KOps/s 1.7921 KOps/s $\textbf{\color{#35bf28}+20.50\%}$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 4.0126ms 2.2105ms 452.3883 Ops/s 469.1312 Ops/s $\color{#d91a1a}-3.57\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 0.9909ms 0.4926ms 2.0299 KOps/s 2.0643 KOps/s $\color{#d91a1a}-1.67\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.7803ms 0.4604ms 2.1721 KOps/s 2.1685 KOps/s $\color{#35bf28}+0.17\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 3.4992ms 2.2802ms 438.5601 Ops/s 439.0793 Ops/s $\color{#d91a1a}-0.12\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 0.8506ms 0.6087ms 1.6430 KOps/s 1.4316 KOps/s $\textbf{\color{#35bf28}+14.77\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 91.8818ms 0.6789ms 1.4730 KOps/s 1.7454 KOps/s $\textbf{\color{#d91a1a}-15.61\%}$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 83.4421ms 5.2779ms 189.4689 Ops/s 194.1432 Ops/s $\color{#d91a1a}-2.41\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] 14.5374ms 12.1248ms 82.4753 Ops/s 82.9046 Ops/s $\color{#d91a1a}-0.52\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] 1.8026ms 1.0549ms 947.9649 Ops/s 909.9193 Ops/s $\color{#35bf28}+4.18\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] 87.4969ms 6.9010ms 144.9075 Ops/s 147.2350 Ops/s $\color{#d91a1a}-1.58\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] 14.6395ms 12.0894ms 82.7168 Ops/s 83.3640 Ops/s $\color{#d91a1a}-0.78\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] 1.8082ms 1.0563ms 946.7031 Ops/s 944.0432 Ops/s $\color{#35bf28}+0.28\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] 83.8242ms 7.0758ms 141.3268 Ops/s 139.3802 Ops/s $\color{#35bf28}+1.40\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] 14.9502ms 12.3860ms 80.7363 Ops/s 81.4621 Ops/s $\color{#d91a1a}-0.89\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] 2.0888ms 1.3591ms 735.7942 Ops/s 704.9272 Ops/s $\color{#35bf28}+4.38\%$

@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 27, 2024 •
edited
Loading

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of GPU Benchmark Tests

Total Benchmarks: 92. Improved: $\large\color{#35bf28}4$. Worsened: $\large\color{#d91a1a}5$.

Expand to view detailed results
Name Max Mean Ops Ops on Repo HEAD Change
test_single 0.1197s 0.1163s 8.5975 Ops/s 8.6205 Ops/s $\color{#d91a1a}-0.27\%$
test_sync 96.0696ms 95.7709ms 10.4416 Ops/s 10.4598 Ops/s $\color{#d91a1a}-0.17\%$
test_async 0.1819s 91.8702ms 10.8849 Ops/s 10.8347 Ops/s $\color{#35bf28}+0.46\%$
test_single_pixels 0.1985s 0.1383s 7.2300 Ops/s 7.6446 Ops/s $\textbf{\color{#d91a1a}-5.42\%}$
test_sync_pixels 83.1087ms 81.5281ms 12.2657 Ops/s 12.1585 Ops/s $\color{#35bf28}+0.88\%$
test_async_pixels 0.1544s 77.2140ms 12.9510 Ops/s 15.7331 Ops/s $\textbf{\color{#d91a1a}-17.68\%}$
test_simple 0.8894s 0.8352s 1.1973 Ops/s 1.1664 Ops/s $\color{#35bf28}+2.64\%$
test_transformed 1.1173s 1.0588s 0.9445 Ops/s 0.9154 Ops/s $\color{#35bf28}+3.18\%$
test_serial 2.5272s 2.4645s 0.4058 Ops/s 0.3966 Ops/s $\color{#35bf28}+2.30\%$
test_parallel 2.1671s 2.0873s 0.4791 Ops/s 0.4740 Ops/s $\color{#35bf28}+1.07\%$
test_step_mdp_speed[True-True-True-True-True] 0.1072ms 32.7082μs 30.5734 KOps/s 30.3322 KOps/s $\color{#35bf28}+0.80\%$
test_step_mdp_speed[True-True-True-True-False] 43.7610μs 19.6714μs 50.8353 KOps/s 51.1013 KOps/s $\color{#d91a1a}-0.52\%$
test_step_mdp_speed[True-True-True-False-True] 37.4300μs 18.8026μs 53.1841 KOps/s 54.2066 KOps/s $\color{#d91a1a}-1.89\%$
test_step_mdp_speed[True-True-True-False-False] 29.2800μs 10.9380μs 91.4242 KOps/s 90.4130 KOps/s $\color{#35bf28}+1.12\%$
test_step_mdp_speed[True-True-False-True-True] 65.8510μs 33.4269μs 29.9160 KOps/s 29.0962 KOps/s $\color{#35bf28}+2.82\%$
test_step_mdp_speed[True-True-False-True-False] 44.8400μs 20.8175μs 48.0364 KOps/s 47.2311 KOps/s $\color{#35bf28}+1.71\%$
test_step_mdp_speed[True-True-False-False-True] 49.2400μs 19.8589μs 50.3552 KOps/s 49.9286 KOps/s $\color{#35bf28}+0.85\%$
test_step_mdp_speed[True-True-False-False-False] 38.3210μs 12.8880μs 77.5918 KOps/s 77.4371 KOps/s $\color{#35bf28}+0.20\%$
test_step_mdp_speed[True-False-True-True-True] 66.7710μs 35.8279μs 27.9112 KOps/s 27.5834 KOps/s $\color{#35bf28}+1.19\%$
test_step_mdp_speed[True-False-True-True-False] 47.9600μs 23.0007μs 43.4770 KOps/s 43.1498 KOps/s $\color{#35bf28}+0.76\%$
test_step_mdp_speed[True-False-True-False-True] 51.6810μs 20.0892μs 49.7781 KOps/s 49.1897 KOps/s $\color{#35bf28}+1.20\%$
test_step_mdp_speed[True-False-True-False-False] 36.4110μs 12.9192μs 77.4044 KOps/s 76.4533 KOps/s $\color{#35bf28}+1.24\%$
test_step_mdp_speed[True-False-False-True-True] 83.7320μs 37.5331μs 26.6432 KOps/s 25.9078 KOps/s $\color{#35bf28}+2.84\%$
test_step_mdp_speed[True-False-False-True-False] 49.3700μs 24.6343μs 40.5938 KOps/s 39.2296 KOps/s $\color{#35bf28}+3.48\%$
test_step_mdp_speed[True-False-False-False-True] 58.6010μs 22.1483μs 45.1501 KOps/s 46.2375 KOps/s $\color{#d91a1a}-2.35\%$
test_step_mdp_speed[True-False-False-False-False] 45.7610μs 14.7255μs 67.9096 KOps/s 67.5911 KOps/s $\color{#35bf28}+0.47\%$
test_step_mdp_speed[False-True-True-True-True] 92.8210μs 35.8524μs 27.8922 KOps/s 27.0643 KOps/s $\color{#35bf28}+3.06\%$
test_step_mdp_speed[False-True-True-True-False] 55.7410μs 22.9670μs 43.5407 KOps/s 42.5914 KOps/s $\color{#35bf28}+2.23\%$
test_step_mdp_speed[False-True-True-False-True] 51.7610μs 23.6010μs 42.3712 KOps/s 41.3042 KOps/s $\color{#35bf28}+2.58\%$
test_step_mdp_speed[False-True-True-False-False] 36.9110μs 14.7746μs 67.6838 KOps/s 66.4299 KOps/s $\color{#35bf28}+1.89\%$
test_step_mdp_speed[False-True-False-True-True] 77.3820μs 37.8654μs 26.4093 KOps/s 25.4218 KOps/s $\color{#35bf28}+3.88\%$
test_step_mdp_speed[False-True-False-True-False] 60.7410μs 25.5538μs 39.1331 KOps/s 38.4341 KOps/s $\color{#35bf28}+1.82\%$
test_step_mdp_speed[False-True-False-False-True] 61.6510μs 25.9748μs 38.4989 KOps/s 37.4128 KOps/s $\color{#35bf28}+2.90\%$
test_step_mdp_speed[False-True-False-False-False] 46.0510μs 16.6188μs 60.1727 KOps/s 57.8502 KOps/s $\color{#35bf28}+4.01\%$
test_step_mdp_speed[False-False-True-True-True] 66.6510μs 40.0073μs 24.9955 KOps/s 23.9469 KOps/s $\color{#35bf28}+4.38\%$
test_step_mdp_speed[False-False-True-True-False] 48.6610μs 27.1393μs 36.8469 KOps/s 36.1599 KOps/s $\color{#35bf28}+1.90\%$
test_step_mdp_speed[False-False-True-False-True] 51.9610μs 25.2675μs 39.5765 KOps/s 38.1678 KOps/s $\color{#35bf28}+3.69\%$
test_step_mdp_speed[False-False-True-False-False] 46.2510μs 16.5512μs 60.4185 KOps/s 58.8860 KOps/s $\color{#35bf28}+2.60\%$
test_step_mdp_speed[False-False-False-True-True] 69.2910μs 41.2098μs 24.2661 KOps/s 23.3573 KOps/s $\color{#35bf28}+3.89\%$
test_step_mdp_speed[False-False-False-True-False] 55.4210μs 28.9215μs 34.5763 KOps/s 33.4372 KOps/s $\color{#35bf28}+3.41\%$
test_step_mdp_speed[False-False-False-False-True] 52.7010μs 27.6230μs 36.2017 KOps/s 35.4049 KOps/s $\color{#35bf28}+2.25\%$
test_step_mdp_speed[False-False-False-False-False] 40.3700μs 18.3526μs 54.4882 KOps/s 52.3527 KOps/s $\color{#35bf28}+4.08\%$
test_values[generalized_advantage_estimate-True-True] 25.9339ms 25.2210ms 39.6495 Ops/s 40.7670 Ops/s $\color{#d91a1a}-2.74\%$
test_values[vec_generalized_advantage_estimate-True-True] 84.6647ms 3.2635ms 306.4213 Ops/s 301.8175 Ops/s $\color{#35bf28}+1.53\%$
test_values[td0_return_estimate-False-False] 96.9020μs 60.3129μs 16.5802 KOps/s 16.9435 KOps/s $\color{#d91a1a}-2.14\%$
test_values[td1_return_estimate-False-False] 55.7105ms 53.8822ms 18.5590 Ops/s 19.2954 Ops/s $\color{#d91a1a}-3.82\%$
test_values[vec_td1_return_estimate-False-False] 1.9890ms 1.7640ms 566.8953 Ops/s 568.7573 Ops/s $\color{#d91a1a}-0.33\%$
test_values[td_lambda_return_estimate-True-False] 88.8871ms 86.5326ms 11.5563 Ops/s 11.4801 Ops/s $\color{#35bf28}+0.66\%$
test_values[vec_td_lambda_return_estimate-True-False] 3.8199ms 1.7922ms 557.9816 Ops/s 560.2772 Ops/s $\color{#d91a1a}-0.41\%$
test_gae_speed[generalized_advantage_estimate-False-1-512] 24.4089ms 23.6592ms 42.2669 Ops/s 44.2776 Ops/s $\color{#d91a1a}-4.54\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512] 0.9143ms 0.7155ms 1.3977 KOps/s 1.4623 KOps/s $\color{#d91a1a}-4.42\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512] 0.8026ms 0.6593ms 1.5167 KOps/s 1.5425 KOps/s $\color{#d91a1a}-1.67\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512] 1.5603ms 1.4614ms 684.2706 Ops/s 692.7310 Ops/s $\color{#d91a1a}-1.22\%$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512] 0.9916ms 0.6883ms 1.4528 KOps/s 1.5219 KOps/s $\color{#d91a1a}-4.54\%$
test_dqn_speed 8.1883ms 1.4905ms 670.9227 Ops/s 653.6460 Ops/s $\color{#35bf28}+2.64\%$
test_ddpg_speed 3.1072ms 2.7627ms 361.9709 Ops/s 358.9984 Ops/s $\color{#35bf28}+0.83\%$
test_sac_speed 81.9151ms 8.6172ms 116.0473 Ops/s 125.8513 Ops/s $\textbf{\color{#d91a1a}-7.79\%}$
test_redq_speed 11.2316ms 10.1747ms 98.2831 Ops/s 96.7165 Ops/s $\color{#35bf28}+1.62\%$
test_redq_deprec_speed 11.6480ms 11.0091ms 90.8340 Ops/s 91.2263 Ops/s $\color{#d91a1a}-0.43\%$
test_td3_speed 8.3927ms 7.9757ms 125.3808 Ops/s 125.4887 Ops/s $\color{#d91a1a}-0.09\%$
test_cql_speed 26.2358ms 25.4412ms 39.3063 Ops/s 39.3745 Ops/s $\color{#d91a1a}-0.17\%$
test_a2c_speed 5.8739ms 5.5191ms 181.1892 Ops/s 180.5996 Ops/s $\color{#35bf28}+0.33\%$
test_ppo_speed 5.9982ms 5.7887ms 172.7506 Ops/s 171.1791 Ops/s $\color{#35bf28}+0.92\%$
test_reinforce_speed 6.1469ms 4.5697ms 218.8350 Ops/s 223.4037 Ops/s $\color{#d91a1a}-2.05\%$
test_iql_speed 19.9472ms 19.3770ms 51.6076 Ops/s 51.1282 Ops/s $\color{#35bf28}+0.94\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 3.5617ms 2.8865ms 346.4387 Ops/s 351.3294 Ops/s $\color{#d91a1a}-1.39\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.6696ms 0.5552ms 1.8012 KOps/s 1.8147 KOps/s $\color{#d91a1a}-0.74\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.7246ms 0.5324ms 1.8783 KOps/s 1.8966 KOps/s $\color{#d91a1a}-0.97\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 3.1763ms 2.8932ms 345.6400 Ops/s 347.9830 Ops/s $\color{#d91a1a}-0.67\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 0.8540ms 0.5458ms 1.8323 KOps/s 1.8359 KOps/s $\color{#d91a1a}-0.20\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 4.3827ms 0.5272ms 1.8969 KOps/s 1.9113 KOps/s $\color{#d91a1a}-0.75\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 3.3082ms 3.0347ms 329.5219 Ops/s 335.0267 Ops/s $\color{#d91a1a}-1.64\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 0.8035ms 0.6853ms 1.4592 KOps/s 1.4876 KOps/s $\color{#d91a1a}-1.91\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 4.2985ms 0.6610ms 1.5128 KOps/s 1.5294 KOps/s $\color{#d91a1a}-1.09\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 3.0864ms 2.8835ms 346.8035 Ops/s 349.3170 Ops/s $\color{#d91a1a}-0.72\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.8133ms 0.5524ms 1.8103 KOps/s 1.8040 KOps/s $\color{#35bf28}+0.35\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 4.2095ms 0.5302ms 1.8861 KOps/s 1.8810 KOps/s $\color{#35bf28}+0.27\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 3.2488ms 2.9148ms 343.0817 Ops/s 342.1059 Ops/s $\color{#35bf28}+0.29\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 0.6595ms 0.5488ms 1.8221 KOps/s 1.8289 KOps/s $\color{#d91a1a}-0.37\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 4.2387ms 0.5266ms 1.8990 KOps/s 1.9137 KOps/s $\color{#d91a1a}-0.77\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 3.3461ms 3.0343ms 329.5684 Ops/s 333.7818 Ops/s $\color{#d91a1a}-1.26\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 0.9742ms 0.6756ms 1.4802 KOps/s 1.2919 KOps/s $\textbf{\color{#35bf28}+14.57\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.1060s 0.7509ms 1.3317 KOps/s 1.5248 KOps/s $\textbf{\color{#d91a1a}-12.66\%}$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 0.1033s 6.7429ms 148.3041 Ops/s 150.2873 Ops/s $\color{#d91a1a}-1.32\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] 17.5692ms 15.1089ms 66.1862 Ops/s 65.1672 Ops/s $\color{#35bf28}+1.56\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] 2.1111ms 1.0827ms 923.5801 Ops/s 830.0409 Ops/s $\textbf{\color{#35bf28}+11.27\%}$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] 0.1012s 8.6182ms 116.0339 Ops/s 116.6915 Ops/s $\color{#d91a1a}-0.56\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] 17.6358ms 15.0454ms 66.4655 Ops/s 66.0502 Ops/s $\color{#35bf28}+0.63\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] 2.1598ms 1.0918ms 915.9190 Ops/s 771.7874 Ops/s $\textbf{\color{#35bf28}+18.68\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] 0.1011s 7.1092ms 140.6625 Ops/s 145.3824 Ops/s $\color{#d91a1a}-3.25\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] 0.1108s 17.3567ms 57.6147 Ops/s 64.9777 Ops/s $\textbf{\color{#d91a1a}-11.33\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] 1.7752ms 1.4297ms 699.4426 Ops/s 625.8402 Ops/s $\textbf{\color{#35bf28}+11.76\%}$

@albertbou92
Copy link
Contributor

Great thanks! The current errors don't seem related to the reduction parameter.

@albertbou92
Copy link
Contributor

I have added the on-policy tests here: #1972

@vmoens vmoens merged commit db4ad23 into main Feb 27, 2024
65 of 68 checks passed
@vmoens vmoens deleted the fix-objectives branch February 27, 2024 16:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
bug Something isn't working CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@vmoens @albertbou92 @facebook-github-bot