Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature] Replay buffer checkpointers #2137

Merged
merged 14 commits into from
May 16, 2024
Merged

[Feature] Replay buffer checkpointers #2137

merged 14 commits into from
May 16, 2024

Conversation

vmoens
Copy link
Contributor

@vmoens vmoens commented Apr 29, 2024 •
edited
Loading

Further work (separate PR):

  • Add checks in replay buffers for checkpointers that require the storage to contain continuous, well defined trajectories

@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Apr 29, 2024 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/2137

Note: Links to docs will display an error until the docs builds have been completed.

❌ 6 New Failures, 5 Unrelated Failures

As of commit 2a184b3 with merge base e77f0dd (image):

NEW FAILURES - The following jobs have failed:

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Apr 29, 2024
@github-actions GitHub Actions
Copy link

github-actions bot commented Apr 29, 2024 •
edited
Loading

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of CPU Benchmark Tests

Total Benchmarks: 91. Improved: $\large\color{#35bf28}10$. Worsened: $\large\color{#d91a1a}2$.

Expand to view detailed results
Name Max Mean Ops Ops on Repo HEAD Change
test_single 54.2698ms 53.8804ms 18.5596 Ops/s 18.1807 Ops/s $\color{#35bf28}+2.08\%$
test_sync 41.4596ms 31.0590ms 32.1967 Ops/s 31.4228 Ops/s $\color{#35bf28}+2.46\%$
test_async 55.0691ms 27.3634ms 36.5451 Ops/s 34.7455 Ops/s $\textbf{\color{#35bf28}+5.18\%}$
test_simple 0.4066s 0.3474s 2.8787 Ops/s 2.9040 Ops/s $\color{#d91a1a}-0.87\%$
test_transformed 0.5407s 0.4884s 2.0475 Ops/s 1.9999 Ops/s $\color{#35bf28}+2.38\%$
test_serial 1.2684s 1.2023s 0.8317 Ops/s 0.7913 Ops/s $\textbf{\color{#35bf28}+5.11\%}$
test_parallel 1.0590s 1.0010s 0.9990 Ops/s 0.9717 Ops/s $\color{#35bf28}+2.80\%$
test_step_mdp_speed[True-True-True-True-True] 0.1748ms 21.5470μs 46.4101 KOps/s 43.5978 KOps/s $\textbf{\color{#35bf28}+6.45\%}$
test_step_mdp_speed[True-True-True-True-False] 58.4810μs 13.2770μs 75.3183 KOps/s 75.3939 KOps/s $\color{#d91a1a}-0.10\%$
test_step_mdp_speed[True-True-True-False-True] 37.7300μs 12.6323μs 79.1622 KOps/s 78.6764 KOps/s $\color{#35bf28}+0.62\%$
test_step_mdp_speed[True-True-True-False-False] 60.4330μs 7.6160μs 131.3033 KOps/s 127.3069 KOps/s $\color{#35bf28}+3.14\%$
test_step_mdp_speed[True-True-False-True-True] 47.0570μs 22.8899μs 43.6875 KOps/s 43.0264 KOps/s $\color{#35bf28}+1.54\%$
test_step_mdp_speed[True-True-False-True-False] 61.7950μs 14.4272μs 69.3133 KOps/s 69.1941 KOps/s $\color{#35bf28}+0.17\%$
test_step_mdp_speed[True-True-False-False-True] 66.9950μs 13.9568μs 71.6499 KOps/s 71.3765 KOps/s $\color{#35bf28}+0.38\%$
test_step_mdp_speed[True-True-False-False-False] 66.3240μs 8.9193μs 112.1170 KOps/s 111.2464 KOps/s $\color{#35bf28}+0.78\%$
test_step_mdp_speed[True-False-True-True-True] 75.0200μs 24.7935μs 40.3331 KOps/s 40.4512 KOps/s $\color{#d91a1a}-0.29\%$
test_step_mdp_speed[True-False-True-True-False] 45.7850μs 16.1646μs 61.8635 KOps/s 62.9713 KOps/s $\color{#d91a1a}-1.76\%$
test_step_mdp_speed[True-False-True-False-True] 46.5560μs 13.9880μs 71.4899 KOps/s 71.5144 KOps/s $\color{#d91a1a}-0.03\%$
test_step_mdp_speed[True-False-True-False-False] 23.2540μs 9.0827μs 110.0993 KOps/s 112.2896 KOps/s $\color{#d91a1a}-1.95\%$
test_step_mdp_speed[True-False-False-True-True] 86.6020μs 25.8567μs 38.6748 KOps/s 38.4073 KOps/s $\color{#35bf28}+0.70\%$
test_step_mdp_speed[True-False-False-True-False] 39.8240μs 17.3111μs 57.7666 KOps/s 58.4621 KOps/s $\color{#d91a1a}-1.19\%$
test_step_mdp_speed[True-False-False-False-True] 61.4550μs 15.2610μs 65.5264 KOps/s 66.0260 KOps/s $\color{#d91a1a}-0.76\%$
test_step_mdp_speed[True-False-False-False-False] 43.5210μs 10.2544μs 97.5191 KOps/s 98.7992 KOps/s $\color{#d91a1a}-1.30\%$
test_step_mdp_speed[False-True-True-True-True] 74.4590μs 24.6354μs 40.5920 KOps/s 39.9266 KOps/s $\color{#35bf28}+1.67\%$
test_step_mdp_speed[False-True-True-True-False] 59.1700μs 16.2131μs 61.6784 KOps/s 62.0847 KOps/s $\color{#d91a1a}-0.65\%$
test_step_mdp_speed[False-True-True-False-True] 39.2630μs 16.2779μs 61.4330 KOps/s 61.3047 KOps/s $\color{#35bf28}+0.21\%$
test_step_mdp_speed[False-True-True-False-False] 56.4450μs 10.2412μs 97.6446 KOps/s 96.7279 KOps/s $\color{#35bf28}+0.95\%$
test_step_mdp_speed[False-True-False-True-True] 59.7020μs 25.4127μs 39.3504 KOps/s 38.7533 KOps/s $\color{#35bf28}+1.54\%$
test_step_mdp_speed[False-True-False-True-False] 55.2730μs 17.1035μs 58.4677 KOps/s 57.7954 KOps/s $\color{#35bf28}+1.16\%$
test_step_mdp_speed[False-True-False-False-True] 44.1720μs 17.4633μs 57.2631 KOps/s 56.5512 KOps/s $\color{#35bf28}+1.26\%$
test_step_mdp_speed[False-True-False-False-False] 72.2950μs 11.5203μs 86.8037 KOps/s 85.7605 KOps/s $\color{#35bf28}+1.22\%$
test_step_mdp_speed[False-False-True-True-True] 86.6810μs 27.1981μs 36.7673 KOps/s 36.6139 KOps/s $\color{#35bf28}+0.42\%$
test_step_mdp_speed[False-False-True-True-False] 47.6390μs 18.5789μs 53.8245 KOps/s 53.4197 KOps/s $\color{#35bf28}+0.76\%$
test_step_mdp_speed[False-False-True-False-True] 65.9630μs 17.2546μs 57.9555 KOps/s 56.9784 KOps/s $\color{#35bf28}+1.71\%$
test_step_mdp_speed[False-False-True-False-False] 34.3240μs 11.4385μs 87.4241 KOps/s 87.9153 KOps/s $\color{#d91a1a}-0.56\%$
test_step_mdp_speed[False-False-False-True-True] 43.7820μs 28.7157μs 34.8241 KOps/s 34.6462 KOps/s $\color{#35bf28}+0.51\%$
test_step_mdp_speed[False-False-False-True-False] 73.1560μs 19.5834μs 51.0636 KOps/s 50.9585 KOps/s $\color{#35bf28}+0.21\%$
test_step_mdp_speed[False-False-False-False-True] 35.3860μs 18.6051μs 53.7486 KOps/s 52.9428 KOps/s $\color{#35bf28}+1.52\%$
test_step_mdp_speed[False-False-False-False-False] 62.2670μs 12.6253μs 79.2060 KOps/s 78.8259 KOps/s $\color{#35bf28}+0.48\%$
test_values[generalized_advantage_estimate-True-True] 9.8038ms 9.3477ms 106.9783 Ops/s 102.5110 Ops/s $\color{#35bf28}+4.36\%$
test_values[vec_generalized_advantage_estimate-True-True] 37.1094ms 33.5259ms 29.8277 Ops/s 29.8811 Ops/s $\color{#d91a1a}-0.18\%$
test_values[td0_return_estimate-False-False] 0.2330ms 0.1804ms 5.5445 KOps/s 5.4880 KOps/s $\color{#35bf28}+1.03\%$
test_values[td1_return_estimate-False-False] 26.1356ms 22.8502ms 43.7632 Ops/s 41.4164 Ops/s $\textbf{\color{#35bf28}+5.67\%}$
test_values[vec_td1_return_estimate-False-False] 35.5232ms 33.6493ms 29.7183 Ops/s 29.7692 Ops/s $\color{#d91a1a}-0.17\%$
test_values[td_lambda_return_estimate-True-False] 35.9929ms 33.1154ms 30.1974 Ops/s 28.5451 Ops/s $\textbf{\color{#35bf28}+5.79\%}$
test_values[vec_td_lambda_return_estimate-True-False] 36.1030ms 33.6655ms 29.7040 Ops/s 29.7782 Ops/s $\color{#d91a1a}-0.25\%$
test_gae_speed[generalized_advantage_estimate-False-1-512] 11.3004ms 8.2385ms 121.3815 Ops/s 116.9440 Ops/s $\color{#35bf28}+3.79\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512] 2.1403ms 1.7758ms 563.1186 Ops/s 529.5992 Ops/s $\textbf{\color{#35bf28}+6.33\%}$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512] 0.4445ms 0.3477ms 2.8760 KOps/s 2.8095 KOps/s $\color{#35bf28}+2.37\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512] 48.0511ms 45.5206ms 21.9681 Ops/s 23.6932 Ops/s $\textbf{\color{#d91a1a}-7.28\%}$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512] 4.8934ms 3.1446ms 318.0062 Ops/s 328.0703 Ops/s $\color{#d91a1a}-3.07\%$
test_dqn_speed 6.3870ms 1.3552ms 737.8909 Ops/s 719.0404 Ops/s $\color{#35bf28}+2.62\%$
test_ddpg_speed 3.7247ms 2.9099ms 343.6555 Ops/s 342.9027 Ops/s $\color{#35bf28}+0.22\%$
test_sac_speed 11.4324ms 8.6781ms 115.2325 Ops/s 105.1648 Ops/s $\textbf{\color{#35bf28}+9.57\%}$
test_redq_speed 0.1004s 14.6201ms 68.3992 Ops/s 74.5903 Ops/s $\textbf{\color{#d91a1a}-8.30\%}$
test_redq_deprec_speed 15.1831ms 14.2596ms 70.1284 Ops/s 70.9278 Ops/s $\color{#d91a1a}-1.13\%$
test_td3_speed 8.8918ms 8.5586ms 116.8409 Ops/s 114.0368 Ops/s $\color{#35bf28}+2.46\%$
test_cql_speed 38.1979ms 36.8355ms 27.1477 Ops/s 26.8119 Ops/s $\color{#35bf28}+1.25\%$
test_a2c_speed 8.7842ms 7.5392ms 132.6401 Ops/s 130.1530 Ops/s $\color{#35bf28}+1.91\%$
test_ppo_speed 13.5535ms 8.1119ms 123.2751 Ops/s 126.3649 Ops/s $\color{#d91a1a}-2.45\%$
test_reinforce_speed 7.7784ms 6.7982ms 147.0976 Ops/s 145.5173 Ops/s $\color{#35bf28}+1.09\%$
test_iql_speed 38.2174ms 33.6702ms 29.6999 Ops/s 29.8851 Ops/s $\color{#d91a1a}-0.62\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 3.3231ms 2.3715ms 421.6810 Ops/s 434.6546 Ops/s $\color{#d91a1a}-2.98\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.8463ms 0.5235ms 1.9104 KOps/s 1.9305 KOps/s $\color{#d91a1a}-1.04\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 3.4924ms 0.4847ms 2.0632 KOps/s 2.0032 KOps/s $\color{#35bf28}+3.00\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 3.9475ms 2.3069ms 433.4815 Ops/s 414.1645 Ops/s $\color{#35bf28}+4.66\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 0.9364ms 0.5071ms 1.9718 KOps/s 1.9664 KOps/s $\color{#35bf28}+0.28\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 1.3122ms 0.4961ms 2.0156 KOps/s 2.0252 KOps/s $\color{#d91a1a}-0.48\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000] 2.0155ms 1.2375ms 808.0866 Ops/s 786.7068 Ops/s $\color{#35bf28}+2.72\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000] 1.6949ms 1.1732ms 852.3914 Ops/s 836.7683 Ops/s $\color{#35bf28}+1.87\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 2.5292ms 2.3644ms 422.9357 Ops/s 413.5734 Ops/s $\color{#35bf28}+2.26\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 0.7556ms 0.6188ms 1.6160 KOps/s 1.5879 KOps/s $\color{#35bf28}+1.77\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 3.6146ms 0.6002ms 1.6661 KOps/s 1.6452 KOps/s $\color{#35bf28}+1.27\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 2.4535ms 2.2600ms 442.4699 Ops/s 435.1385 Ops/s $\color{#35bf28}+1.68\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.8308ms 0.5067ms 1.9734 KOps/s 1.9374 KOps/s $\color{#35bf28}+1.86\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.7758ms 0.4853ms 2.0604 KOps/s 1.8778 KOps/s $\textbf{\color{#35bf28}+9.73\%}$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 3.8922ms 2.3108ms 432.7457 Ops/s 431.5418 Ops/s $\color{#35bf28}+0.28\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 1.1090ms 0.5011ms 1.9957 KOps/s 1.9547 KOps/s $\color{#35bf28}+2.10\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 1.0926ms 0.4913ms 2.0356 KOps/s 1.9800 KOps/s $\color{#35bf28}+2.81\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 2.5359ms 2.3471ms 426.0551 Ops/s 411.8816 Ops/s $\color{#35bf28}+3.44\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 0.8995ms 0.6263ms 1.5966 KOps/s 1.5762 KOps/s $\color{#35bf28}+1.29\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.9175ms 0.5987ms 1.6703 KOps/s 1.6450 KOps/s $\color{#35bf28}+1.54\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 0.1199s 8.2055ms 121.8701 Ops/s 121.2272 Ops/s $\color{#35bf28}+0.53\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] 15.3006ms 12.7320ms 78.5421 Ops/s 76.1609 Ops/s $\color{#35bf28}+3.13\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] 1.7119ms 1.0764ms 929.0355 Ops/s 823.5155 Ops/s $\textbf{\color{#35bf28}+12.81\%}$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] 0.1072s 5.7720ms 173.2514 Ops/s 170.2512 Ops/s $\color{#35bf28}+1.76\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] 14.8745ms 12.6689ms 78.9333 Ops/s 76.7655 Ops/s $\color{#35bf28}+2.82\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] 1.8506ms 1.0907ms 916.8477 Ops/s 814.2885 Ops/s $\textbf{\color{#35bf28}+12.59\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] 0.1119s 8.2437ms 121.3042 Ops/s 119.0641 Ops/s $\color{#35bf28}+1.88\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] 16.9090ms 13.1295ms 76.1646 Ops/s 74.8992 Ops/s $\color{#35bf28}+1.69\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] 2.1344ms 1.3791ms 725.0892 Ops/s 698.9755 Ops/s $\color{#35bf28}+3.74\%$

@github-actions GitHub Actions
Copy link

github-actions bot commented Apr 29, 2024 •
edited
Loading

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of GPU Benchmark Tests

Total Benchmarks: 94. Improved: $\large\color{#35bf28}3$. Worsened: $\large\color{#d91a1a}5$.

Expand to view detailed results
Name Max Mean Ops Ops on Repo HEAD Change
test_single 0.1160s 0.1158s 8.6367 Ops/s 8.5706 Ops/s $\color{#35bf28}+0.77\%$
test_sync 0.1023s 0.1015s 9.8479 Ops/s 9.8118 Ops/s $\color{#35bf28}+0.37\%$
test_async 0.1914s 95.0424ms 10.5216 Ops/s 10.3051 Ops/s $\color{#35bf28}+2.10\%$
test_single_pixels 0.1279s 0.1267s 7.8897 Ops/s 7.8849 Ops/s $\color{#35bf28}+0.06\%$
test_sync_pixels 85.0584ms 81.8755ms 12.2137 Ops/s 12.2706 Ops/s $\color{#d91a1a}-0.46\%$
test_async_pixels 0.1542s 65.6919ms 15.2226 Ops/s 12.3883 Ops/s $\textbf{\color{#35bf28}+22.88\%}$
test_simple 0.8325s 0.7778s 1.2857 Ops/s 1.2906 Ops/s $\color{#d91a1a}-0.38\%$
test_transformed 1.0831s 1.0241s 0.9764 Ops/s 0.9747 Ops/s $\color{#35bf28}+0.18\%$
test_serial 2.5257s 2.4750s 0.4040 Ops/s 0.4047 Ops/s $\color{#d91a1a}-0.17\%$
test_parallel 2.3280s 2.3032s 0.4342 Ops/s 0.4332 Ops/s $\color{#35bf28}+0.24\%$
test_step_mdp_speed[True-True-True-True-True] 96.2820μs 33.3931μs 29.9463 KOps/s 30.5849 KOps/s $\color{#d91a1a}-2.09\%$
test_step_mdp_speed[True-True-True-True-False] 41.9510μs 19.8931μs 50.2687 KOps/s 50.6029 KOps/s $\color{#d91a1a}-0.66\%$
test_step_mdp_speed[True-True-True-False-True] 39.4700μs 18.7494μs 53.3350 KOps/s 53.5147 KOps/s $\color{#d91a1a}-0.34\%$
test_step_mdp_speed[True-True-True-False-False] 36.9410μs 11.3721μs 87.9344 KOps/s 89.2989 KOps/s $\color{#d91a1a}-1.53\%$
test_step_mdp_speed[True-True-False-True-True] 50.5920μs 35.2637μs 28.3577 KOps/s 28.8091 KOps/s $\color{#d91a1a}-1.57\%$
test_step_mdp_speed[True-True-False-True-False] 40.8800μs 21.8100μs 45.8504 KOps/s 46.0118 KOps/s $\color{#d91a1a}-0.35\%$
test_step_mdp_speed[True-True-False-False-True] 41.6120μs 20.6315μs 48.4695 KOps/s 48.4935 KOps/s $\color{#d91a1a}-0.05\%$
test_step_mdp_speed[True-True-False-False-False] 28.0610μs 13.1862μs 75.8370 KOps/s 76.3053 KOps/s $\color{#d91a1a}-0.61\%$
test_step_mdp_speed[True-False-True-True-True] 64.1520μs 37.3944μs 26.7420 KOps/s 27.0942 KOps/s $\color{#d91a1a}-1.30\%$
test_step_mdp_speed[True-False-True-True-False] 52.3310μs 23.6431μs 42.2956 KOps/s 42.5157 KOps/s $\color{#d91a1a}-0.52\%$
test_step_mdp_speed[True-False-True-False-True] 38.4120μs 20.6728μs 48.3727 KOps/s 47.7711 KOps/s $\color{#35bf28}+1.26\%$
test_step_mdp_speed[True-False-True-False-False] 31.4910μs 13.2720μs 75.3463 KOps/s 75.7364 KOps/s $\color{#d91a1a}-0.52\%$
test_step_mdp_speed[True-False-False-True-True] 65.8610μs 38.8919μs 25.7123 KOps/s 25.8796 KOps/s $\color{#d91a1a}-0.65\%$
test_step_mdp_speed[True-False-False-True-False] 96.0820μs 25.5997μs 39.0630 KOps/s 39.3394 KOps/s $\color{#d91a1a}-0.70\%$
test_step_mdp_speed[True-False-False-False-True] 49.5510μs 22.4488μs 44.5458 KOps/s 45.0086 KOps/s $\color{#d91a1a}-1.03\%$
test_step_mdp_speed[True-False-False-False-False] 31.7210μs 15.1653μs 65.9400 KOps/s 67.2557 KOps/s $\color{#d91a1a}-1.96\%$
test_step_mdp_speed[False-True-True-True-True] 55.1710μs 37.2965μs 26.8122 KOps/s 27.0783 KOps/s $\color{#d91a1a}-0.98\%$
test_step_mdp_speed[False-True-True-True-False] 42.8610μs 23.7071μs 42.1815 KOps/s 42.1439 KOps/s $\color{#35bf28}+0.09\%$
test_step_mdp_speed[False-True-True-False-True] 40.7400μs 24.8752μs 40.2006 KOps/s 40.8364 KOps/s $\color{#d91a1a}-1.56\%$
test_step_mdp_speed[False-True-True-False-False] 34.9100μs 15.0851μs 66.2906 KOps/s 67.2354 KOps/s $\color{#d91a1a}-1.41\%$
test_step_mdp_speed[False-True-False-True-True] 65.4910μs 38.7929μs 25.7779 KOps/s 25.6723 KOps/s $\color{#35bf28}+0.41\%$
test_step_mdp_speed[False-True-False-True-False] 41.5510μs 25.5755μs 39.0999 KOps/s 39.5501 KOps/s $\color{#d91a1a}-1.14\%$
test_step_mdp_speed[False-True-False-False-True] 43.8210μs 26.4008μs 37.8777 KOps/s 38.0910 KOps/s $\color{#d91a1a}-0.56\%$
test_step_mdp_speed[False-True-False-False-False] 38.7310μs 17.0030μs 58.8131 KOps/s 59.1732 KOps/s $\color{#d91a1a}-0.61\%$
test_step_mdp_speed[False-False-True-True-True] 60.1410μs 41.1495μs 24.3016 KOps/s 24.6983 KOps/s $\color{#d91a1a}-1.61\%$
test_step_mdp_speed[False-False-True-True-False] 85.3310μs 27.7281μs 36.0645 KOps/s 36.7163 KOps/s $\color{#d91a1a}-1.78\%$
test_step_mdp_speed[False-False-True-False-True] 52.9210μs 26.8747μs 37.2098 KOps/s 38.1930 KOps/s $\color{#d91a1a}-2.57\%$
test_step_mdp_speed[False-False-True-False-False] 39.4710μs 17.0234μs 58.7427 KOps/s 60.3084 KOps/s $\color{#d91a1a}-2.60\%$
test_step_mdp_speed[False-False-False-True-True] 65.6410μs 42.6143μs 23.4663 KOps/s 23.5304 KOps/s $\color{#d91a1a}-0.27\%$
test_step_mdp_speed[False-False-False-True-False] 52.5610μs 29.4425μs 33.9645 KOps/s 34.2799 KOps/s $\color{#d91a1a}-0.92\%$
test_step_mdp_speed[False-False-False-False-True] 43.7010μs 28.1828μs 35.4826 KOps/s 36.0431 KOps/s $\color{#d91a1a}-1.56\%$
test_step_mdp_speed[False-False-False-False-False] 43.6700μs 18.5376μs 53.9445 KOps/s 54.4524 KOps/s $\color{#d91a1a}-0.93\%$
test_values[generalized_advantage_estimate-True-True] 25.3544ms 24.9137ms 40.1386 Ops/s 40.0983 Ops/s $\color{#35bf28}+0.10\%$
test_values[vec_generalized_advantage_estimate-True-True] 87.5329ms 3.3209ms 301.1218 Ops/s 314.2743 Ops/s $\color{#d91a1a}-4.19\%$
test_values[td0_return_estimate-False-False] 92.3420μs 64.0622μs 15.6098 KOps/s 15.6764 KOps/s $\color{#d91a1a}-0.42\%$
test_values[td1_return_estimate-False-False] 53.8513ms 53.3629ms 18.7396 Ops/s 18.7358 Ops/s $\color{#35bf28}+0.02\%$
test_values[vec_td1_return_estimate-False-False] 2.0813ms 1.7682ms 565.5353 Ops/s 565.5376 Ops/s $-0.00\%$
test_values[td_lambda_return_estimate-True-False] 85.4243ms 84.9818ms 11.7672 Ops/s 11.7206 Ops/s $\color{#35bf28}+0.40\%$
test_values[vec_td_lambda_return_estimate-True-False] 2.1360ms 1.7650ms 566.5825 Ops/s 565.6951 Ops/s $\color{#35bf28}+0.16\%$
test_gae_speed[generalized_advantage_estimate-False-1-512] 23.9492ms 23.7315ms 42.1381 Ops/s 41.9107 Ops/s $\color{#35bf28}+0.54\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512] 0.8909ms 0.6993ms 1.4300 KOps/s 1.4289 KOps/s $\color{#35bf28}+0.08\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512] 0.7236ms 0.6529ms 1.5316 KOps/s 1.5201 KOps/s $\color{#35bf28}+0.75\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512] 1.4906ms 1.4548ms 687.3836 Ops/s 687.8456 Ops/s $\color{#d91a1a}-0.07\%$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512] 0.9314ms 0.6695ms 1.4936 KOps/s 1.4845 KOps/s $\color{#35bf28}+0.61\%$
test_dqn_speed 1.8216ms 1.4736ms 678.6066 Ops/s 690.9173 Ops/s $\color{#d91a1a}-1.78\%$
test_ddpg_speed 3.3253ms 3.0054ms 332.7397 Ops/s 337.0056 Ops/s $\color{#d91a1a}-1.27\%$
test_sac_speed 8.8732ms 8.6140ms 116.0897 Ops/s 117.5075 Ops/s $\color{#d91a1a}-1.21\%$
test_redq_speed 11.3136ms 10.5434ms 94.8461 Ops/s 97.8302 Ops/s $\color{#d91a1a}-3.05\%$
test_redq_deprec_speed 12.1959ms 11.6095ms 86.1366 Ops/s 85.8475 Ops/s $\color{#35bf28}+0.34\%$
test_td3_speed 17.1526ms 8.6951ms 115.0074 Ops/s 118.8552 Ops/s $\color{#d91a1a}-3.24\%$
test_cql_speed 27.5573ms 26.5550ms 37.6577 Ops/s 38.6885 Ops/s $\color{#d91a1a}-2.66\%$
test_a2c_speed 5.9792ms 5.7922ms 172.6470 Ops/s 174.8075 Ops/s $\color{#d91a1a}-1.24\%$
test_ppo_speed 6.2594ms 6.0622ms 164.9578 Ops/s 166.7314 Ops/s $\color{#d91a1a}-1.06\%$
test_reinforce_speed 4.9918ms 4.7654ms 209.8445 Ops/s 216.4252 Ops/s $\color{#d91a1a}-3.04\%$
test_iql_speed 21.0907ms 20.3950ms 49.0316 Ops/s 50.2914 Ops/s $\color{#d91a1a}-2.50\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 3.0602ms 2.8048ms 356.5346 Ops/s 362.6844 Ops/s $\color{#d91a1a}-1.70\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.7324ms 0.6031ms 1.6581 KOps/s 1.6699 KOps/s $\color{#d91a1a}-0.71\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 4.3135ms 0.5785ms 1.7286 KOps/s 1.7423 KOps/s $\color{#d91a1a}-0.79\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 3.1331ms 2.8344ms 352.8131 Ops/s 361.0986 Ops/s $\color{#d91a1a}-2.29\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 0.7078ms 0.5943ms 1.6826 KOps/s 1.6993 KOps/s $\color{#d91a1a}-0.98\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 4.2340ms 0.5736ms 1.7435 KOps/s 1.7554 KOps/s $\color{#d91a1a}-0.68\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000] 1.7487ms 1.5826ms 631.8858 Ops/s 647.7757 Ops/s $\color{#d91a1a}-2.45\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000] 1.6338ms 1.4960ms 668.4339 Ops/s 680.8206 Ops/s $\color{#d91a1a}-1.82\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 3.1023ms 2.9517ms 338.7827 Ops/s 343.1340 Ops/s $\color{#d91a1a}-1.27\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 1.2772ms 0.7368ms 1.3573 KOps/s 1.3717 KOps/s $\color{#d91a1a}-1.05\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.1147s 0.8156ms 1.2261 KOps/s 1.4151 KOps/s $\textbf{\color{#d91a1a}-13.35\%}$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 2.9303ms 2.8282ms 353.5761 Ops/s 358.1402 Ops/s $\color{#d91a1a}-1.27\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.7316ms 0.6040ms 1.6556 KOps/s 1.6727 KOps/s $\color{#d91a1a}-1.02\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 4.4461ms 0.5836ms 1.7135 KOps/s 1.7311 KOps/s $\color{#d91a1a}-1.01\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 3.0297ms 2.8511ms 350.7400 Ops/s 355.2399 Ops/s $\color{#d91a1a}-1.27\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 0.1144s 0.7800ms 1.2820 KOps/s 1.6830 KOps/s $\textbf{\color{#d91a1a}-23.83\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.7218ms 0.5700ms 1.7542 KOps/s 1.7547 KOps/s $\color{#d91a1a}-0.03\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 3.0578ms 2.9752ms 336.1158 Ops/s 340.8276 Ops/s $\color{#d91a1a}-1.38\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 0.8294ms 0.7377ms 1.3556 KOps/s 1.3702 KOps/s $\color{#d91a1a}-1.07\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 4.5973ms 0.7170ms 1.3947 KOps/s 1.4144 KOps/s $\color{#d91a1a}-1.40\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 0.1158s 7.1636ms 139.5955 Ops/s 104.3140 Ops/s $\textbf{\color{#35bf28}+33.82\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] 0.1239s 18.0113ms 55.5206 Ops/s 64.1727 Ops/s $\textbf{\color{#d91a1a}-13.48\%}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] 2.3040ms 1.2550ms 796.8359 Ops/s 855.8339 Ops/s $\textbf{\color{#d91a1a}-6.89\%}$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] 0.1139s 7.1707ms 139.4560 Ops/s 139.7709 Ops/s $\color{#d91a1a}-0.23\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] 18.1749ms 15.8805ms 62.9702 Ops/s 64.2679 Ops/s $\color{#d91a1a}-2.02\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] 2.3344ms 1.2385ms 807.4166 Ops/s 854.3269 Ops/s $\textbf{\color{#d91a1a}-5.49\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] 0.1130s 7.5053ms 133.2385 Ops/s 133.8682 Ops/s $\color{#d91a1a}-0.47\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] 18.7200ms 16.2254ms 61.6318 Ops/s 55.3482 Ops/s $\textbf{\color{#35bf28}+11.35\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] 2.5903ms 1.5831ms 631.6668 Ops/s 614.9586 Ops/s $\color{#35bf28}+2.72\%$

@vmoens vmoens changed the title [Feature] (Un)Flatten data during RB (de)serialization [Feature] Replay buffer checkpointers May 7, 2024
@vmoens vmoens added the enhancement New feature or request label May 16, 2024
Miffyli
Miffyli reviewed May 16, 2024
View reviewed changes
test/test_rb.py Outdated Show resolved Hide resolved
@vmoens vmoens merged commit 73d09c3 into main May 16, 2024
45 of 56 checks passed
@vmoens vmoens deleted the register-data-hooks branch May 16, 2024 15:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Miffyli Miffyli Miffyli left review comments

Assignees
No one assigned
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@vmoens @Miffyli @facebook-github-bot