Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature] Dynamic specs #2143

Merged
merged 27 commits into from
May 31, 2024
Merged

[Feature] Dynamic specs #2143

merged 27 commits into from
May 31, 2024

Conversation

vmoens
Copy link
Contributor

@vmoens vmoens commented Apr 30, 2024 •
edited
Loading

Implements specs with negative shape dimension, allowing these dimensions to account for tensors of variable sizes.

This PR provides basic functionality for specs, environments, batched envs (serial and parallel) and collectors (Sync, MultiSync, MultiaSync).

Performance without buffers in multiprocessed settings isn't great. Funnily, using "cuda" devices to pass data without shared mem buffers is faster than serializing and deserializing tensors (@albanD @mikaylagawarecki please don't make fun of me):

if __name__ == "__main__":
    # this executes a step on cpu on the worker, puts the data on cuda, sends the data through the pipe, and maps it back to cpu afterwards 
    env = ParallelEnv(16, lambda: GymEnv("CartPole-v1", device="cuda"), use_buffers=False, device="cpu")
    timer = torch.utils.benchmark.Timer("env.rollout(200, break_when_any_done=False, return_contiguous=False)")
    print(timer.adaptive_autorange())

I also played a bit in rewriting the reducing methods to tensordict to see if there was any opportunity to make the unpickling faster through some hacky multithreading, without success.

cc @dtsaras @matteobettini

@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Apr 30, 2024 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/2143

Note: Links to docs will display an error until the docs builds have been completed.

❌ 3 New Failures, 4 Unrelated Failures

As of commit 8a30f71 with merge base 765952a (image):

NEW FAILURES - The following jobs have failed:

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Apr 30, 2024
@github-actions GitHub Actions
Copy link

github-actions bot commented May 26, 2024 •
edited
Loading

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of CPU Benchmark Tests

Total Benchmarks: 91. Improved: $\large\color{#35bf28}5$. Worsened: $\large\color{#d91a1a}4$.

Expand to view detailed results
Name Max Mean Ops Ops on Repo HEAD Change
test_single 0.1081s 58.4913ms 17.0966 Ops/s 18.0812 Ops/s $\textbf{\color{#d91a1a}-5.45\%}$
test_sync 38.5727ms 30.7797ms 32.4890 Ops/s 33.2030 Ops/s $\color{#d91a1a}-2.15\%$
test_async 54.5546ms 27.8200ms 35.9453 Ops/s 33.1740 Ops/s $\textbf{\color{#35bf28}+8.35\%}$
test_simple 0.3706s 0.3686s 2.7132 Ops/s 2.7841 Ops/s $\color{#d91a1a}-2.55\%$
test_transformed 0.5279s 0.5227s 1.9133 Ops/s 1.9352 Ops/s $\color{#d91a1a}-1.13\%$
test_serial 1.3019s 1.2532s 0.7980 Ops/s 0.8106 Ops/s $\color{#d91a1a}-1.56\%$
test_parallel 1.1114s 1.0445s 0.9574 Ops/s 0.9576 Ops/s $\color{#d91a1a}-0.02\%$
test_step_mdp_speed[True-True-True-True-True] 0.1559ms 21.7126μs 46.0562 KOps/s 46.8059 KOps/s $\color{#d91a1a}-1.60\%$
test_step_mdp_speed[True-True-True-True-False] 37.6000μs 12.9989μs 76.9298 KOps/s 76.9902 KOps/s $\color{#d91a1a}-0.08\%$
test_step_mdp_speed[True-True-True-False-True] 38.2620μs 12.8861μs 77.6033 KOps/s 78.9798 KOps/s $\color{#d91a1a}-1.74\%$
test_step_mdp_speed[True-True-True-False-False] 18.5550μs 7.6244μs 131.1572 KOps/s 131.0839 KOps/s $\color{#35bf28}+0.06\%$
test_step_mdp_speed[True-True-False-True-True] 57.3480μs 22.9072μs 43.6545 KOps/s 43.9673 KOps/s $\color{#d91a1a}-0.71\%$
test_step_mdp_speed[True-True-False-True-False] 52.0480μs 14.3278μs 69.7942 KOps/s 69.8355 KOps/s $\color{#d91a1a}-0.06\%$
test_step_mdp_speed[True-True-False-False-True] 42.9200μs 13.9969μs 71.4442 KOps/s 72.0613 KOps/s $\color{#d91a1a}-0.86\%$
test_step_mdp_speed[True-True-False-False-False] 34.7160μs 8.9247μs 112.0486 KOps/s 112.2480 KOps/s $\color{#d91a1a}-0.18\%$
test_step_mdp_speed[True-False-True-True-True] 87.1940μs 24.1144μs 41.4691 KOps/s 41.4087 KOps/s $\color{#35bf28}+0.15\%$
test_step_mdp_speed[True-False-True-True-False] 50.6150μs 15.7872μs 63.3425 KOps/s 64.0577 KOps/s $\color{#d91a1a}-1.12\%$
test_step_mdp_speed[True-False-True-False-True] 44.9850μs 13.9066μs 71.9083 KOps/s 71.3726 KOps/s $\color{#35bf28}+0.75\%$
test_step_mdp_speed[True-False-True-False-False] 31.9200μs 8.9216μs 112.0872 KOps/s 111.6787 KOps/s $\color{#35bf28}+0.37\%$
test_step_mdp_speed[True-False-False-True-True] 58.0200μs 25.4825μs 39.2426 KOps/s 39.2807 KOps/s $\color{#d91a1a}-0.10\%$
test_step_mdp_speed[True-False-False-True-False] 54.0920μs 16.6949μs 59.8985 KOps/s 59.5001 KOps/s $\color{#35bf28}+0.67\%$
test_step_mdp_speed[True-False-False-False-True] 48.3310μs 15.1206μs 66.1351 KOps/s 66.6087 KOps/s $\color{#d91a1a}-0.71\%$
test_step_mdp_speed[True-False-False-False-False] 0.1728ms 10.3805μs 96.3343 KOps/s 99.9913 KOps/s $\color{#d91a1a}-3.66\%$
test_step_mdp_speed[False-True-True-True-True] 58.6310μs 24.4346μs 40.9256 KOps/s 41.5218 KOps/s $\color{#d91a1a}-1.44\%$
test_step_mdp_speed[False-True-True-True-False] 38.2120μs 15.7080μs 63.6617 KOps/s 63.6183 KOps/s $\color{#35bf28}+0.07\%$
test_step_mdp_speed[False-True-True-False-True] 72.8280μs 16.2036μs 61.7145 KOps/s 61.9199 KOps/s $\color{#d91a1a}-0.33\%$
test_step_mdp_speed[False-True-True-False-False] 36.2080μs 10.1659μs 98.3680 KOps/s 99.0800 KOps/s $\color{#d91a1a}-0.72\%$
test_step_mdp_speed[False-True-False-True-True] 59.9630μs 25.3196μs 39.4951 KOps/s 39.5295 KOps/s $\color{#d91a1a}-0.09\%$
test_step_mdp_speed[False-True-False-True-False] 0.2915ms 16.9150μs 59.1191 KOps/s 59.4650 KOps/s $\color{#d91a1a}-0.58\%$
test_step_mdp_speed[False-True-False-False-True] 0.1532ms 17.3478μs 57.6442 KOps/s 57.5525 KOps/s $\color{#35bf28}+0.16\%$
test_step_mdp_speed[False-True-False-False-False] 39.4650μs 11.2461μs 88.9196 KOps/s 88.9173 KOps/s $+0.00\%$
test_step_mdp_speed[False-False-True-True-True] 54.1520μs 26.6249μs 37.5588 KOps/s 37.5447 KOps/s $\color{#35bf28}+0.04\%$
test_step_mdp_speed[False-False-True-True-False] 41.6190μs 18.0135μs 55.5140 KOps/s 55.8178 KOps/s $\color{#d91a1a}-0.54\%$
test_step_mdp_speed[False-False-True-False-True] 49.0820μs 17.2696μs 57.9051 KOps/s 57.7408 KOps/s $\color{#35bf28}+0.28\%$
test_step_mdp_speed[False-False-True-False-False] 46.9090μs 11.2646μs 88.7738 KOps/s 89.5853 KOps/s $\color{#d91a1a}-0.91\%$
test_step_mdp_speed[False-False-False-True-True] 38.1120μs 27.9435μs 35.7865 KOps/s 35.6295 KOps/s $\color{#35bf28}+0.44\%$
test_step_mdp_speed[False-False-False-True-False] 60.8370μs 19.0519μs 52.4883 KOps/s 52.4945 KOps/s $\color{#d91a1a}-0.01\%$
test_step_mdp_speed[False-False-False-False-True] 44.7040μs 18.1330μs 55.1480 KOps/s 54.8881 KOps/s $\color{#35bf28}+0.47\%$
test_step_mdp_speed[False-False-False-False-False] 44.9750μs 12.2919μs 81.3543 KOps/s 81.1410 KOps/s $\color{#35bf28}+0.26\%$
test_values[generalized_advantage_estimate-True-True] 10.2165ms 9.6800ms 103.3056 Ops/s 107.5888 Ops/s $\color{#d91a1a}-3.98\%$
test_values[vec_generalized_advantage_estimate-True-True] 49.5255ms 33.7741ms 29.6085 Ops/s 28.1634 Ops/s $\textbf{\color{#35bf28}+5.13\%}$
test_values[td0_return_estimate-False-False] 0.2316ms 0.1648ms 6.0682 KOps/s 6.0955 KOps/s $\color{#d91a1a}-0.45\%$
test_values[td1_return_estimate-False-False] 26.9275ms 23.7937ms 42.0279 Ops/s 43.5647 Ops/s $\color{#d91a1a}-3.53\%$
test_values[vec_td1_return_estimate-False-False] 35.3906ms 33.4504ms 29.8950 Ops/s 27.6924 Ops/s $\textbf{\color{#35bf28}+7.95\%}$
test_values[td_lambda_return_estimate-True-False] 38.1453ms 34.8530ms 28.6920 Ops/s 29.9532 Ops/s $\color{#d91a1a}-4.21\%$
test_values[vec_td_lambda_return_estimate-True-False] 35.5026ms 33.5586ms 29.7986 Ops/s 27.9930 Ops/s $\textbf{\color{#35bf28}+6.45\%}$
test_gae_speed[generalized_advantage_estimate-False-1-512] 8.8294ms 8.4321ms 118.5944 Ops/s 119.0114 Ops/s $\color{#d91a1a}-0.35\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512] 2.3315ms 2.0348ms 491.4388 Ops/s 522.1563 Ops/s $\textbf{\color{#d91a1a}-5.88\%}$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512] 0.4197ms 0.3552ms 2.8153 KOps/s 2.8810 KOps/s $\color{#d91a1a}-2.28\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512] 49.2360ms 45.2392ms 22.1047 Ops/s 23.1565 Ops/s $\color{#d91a1a}-4.54\%$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512] 3.6881ms 3.0213ms 330.9780 Ops/s 330.4490 Ops/s $\color{#35bf28}+0.16\%$
test_dqn_speed 6.6176ms 1.3871ms 720.9279 Ops/s 744.5502 Ops/s $\color{#d91a1a}-3.17\%$
test_ddpg_speed 3.7677ms 2.8270ms 353.7330 Ops/s 350.2750 Ops/s $\color{#35bf28}+0.99\%$
test_sac_speed 8.7422ms 8.3550ms 119.6885 Ops/s 119.9636 Ops/s $\color{#d91a1a}-0.23\%$
test_redq_speed 14.5335ms 13.1957ms 75.7823 Ops/s 73.8880 Ops/s $\color{#35bf28}+2.56\%$
test_redq_deprec_speed 14.1472ms 13.2667ms 75.3766 Ops/s 72.9323 Ops/s $\color{#35bf28}+3.35\%$
test_td3_speed 8.4684ms 8.2756ms 120.8366 Ops/s 120.3183 Ops/s $\color{#35bf28}+0.43\%$
test_cql_speed 37.8813ms 36.4451ms 27.4385 Ops/s 27.3334 Ops/s $\color{#35bf28}+0.38\%$
test_a2c_speed 8.5509ms 7.4585ms 134.0749 Ops/s 133.3058 Ops/s $\color{#35bf28}+0.58\%$
test_ppo_speed 8.2357ms 7.7105ms 129.6939 Ops/s 130.3648 Ops/s $\color{#d91a1a}-0.51\%$
test_reinforce_speed 7.8083ms 6.6603ms 150.1423 Ops/s 151.0233 Ops/s $\color{#d91a1a}-0.58\%$
test_iql_speed 33.7137ms 32.9654ms 30.3349 Ops/s 30.5595 Ops/s $\color{#d91a1a}-0.74\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 5.5176ms 3.5857ms 278.8872 Ops/s 291.7358 Ops/s $\color{#d91a1a}-4.40\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 1.1665ms 0.5056ms 1.9778 KOps/s 2.0102 KOps/s $\color{#d91a1a}-1.61\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.6303ms 0.4751ms 2.1047 KOps/s 2.1013 KOps/s $\color{#35bf28}+0.16\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 5.3601ms 3.5018ms 285.5690 Ops/s 288.5917 Ops/s $\color{#d91a1a}-1.05\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 0.8531ms 0.4964ms 2.0147 KOps/s 2.0392 KOps/s $\color{#d91a1a}-1.20\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 97.9085ms 0.5506ms 1.8162 KOps/s 2.1445 KOps/s $\textbf{\color{#d91a1a}-15.31\%}$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000] 1.4178ms 1.2456ms 802.8113 Ops/s 808.7990 Ops/s $\color{#d91a1a}-0.74\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000] 4.5163ms 1.1862ms 843.0571 Ops/s 758.9590 Ops/s $\textbf{\color{#35bf28}+11.08\%}$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 3.6872ms 3.4924ms 286.3368 Ops/s 282.2494 Ops/s $\color{#35bf28}+1.45\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 1.1487ms 0.6127ms 1.6321 KOps/s 1.6271 KOps/s $\color{#35bf28}+0.31\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.9141ms 0.5858ms 1.7071 KOps/s 1.6839 KOps/s $\color{#35bf28}+1.38\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 11.3639ms 3.8615ms 258.9699 Ops/s 290.8443 Ops/s $\textbf{\color{#d91a1a}-10.96\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 1.1925ms 0.4966ms 2.0137 KOps/s 1.9998 KOps/s $\color{#35bf28}+0.69\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.7192ms 0.4739ms 2.1102 KOps/s 2.0785 KOps/s $\color{#35bf28}+1.53\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 5.2185ms 3.3786ms 295.9827 Ops/s 290.7570 Ops/s $\color{#35bf28}+1.80\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 0.6001ms 0.4873ms 2.0519 KOps/s 2.0036 KOps/s $\color{#35bf28}+2.41\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.7568ms 0.4637ms 2.1563 KOps/s 2.1324 KOps/s $\color{#35bf28}+1.12\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 3.6426ms 3.5339ms 282.9770 Ops/s 279.1760 Ops/s $\color{#35bf28}+1.36\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 1.1554ms 0.6174ms 1.6197 KOps/s 1.6239 KOps/s $\color{#d91a1a}-0.26\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 4.1011ms 0.5898ms 1.6954 KOps/s 1.6801 KOps/s $\color{#35bf28}+0.91\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 98.3464ms 5.6239ms 177.8134 Ops/s 170.6999 Ops/s $\color{#35bf28}+4.17\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] 14.6924ms 12.3142ms 81.2074 Ops/s 80.2214 Ops/s $\color{#35bf28}+1.23\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] 1.7704ms 1.0691ms 935.3283 Ops/s 917.5886 Ops/s $\color{#35bf28}+1.93\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] 91.3531ms 7.1740ms 139.3926 Ops/s 134.5166 Ops/s $\color{#35bf28}+3.62\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] 14.9768ms 12.3098ms 81.2362 Ops/s 80.9240 Ops/s $\color{#35bf28}+0.39\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] 1.5361ms 1.0694ms 935.1167 Ops/s 893.2441 Ops/s $\color{#35bf28}+4.69\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] 91.8691ms 5.7902ms 172.7066 Ops/s 169.7180 Ops/s $\color{#35bf28}+1.76\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] 20.7279ms 12.8658ms 77.7252 Ops/s 78.0764 Ops/s $\color{#d91a1a}-0.45\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] 1.8784ms 1.3753ms 727.1333 Ops/s 711.9895 Ops/s $\color{#35bf28}+2.13\%$

@github-actions GitHub Actions
Copy link

github-actions bot commented May 26, 2024 •
edited
Loading

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of GPU Benchmark Tests

Total Benchmarks: 94. Improved: $\large\color{#35bf28}4$. Worsened: $\large\color{#d91a1a}2$.

Expand to view detailed results
Name Max Mean Ops Ops on Repo HEAD Change
test_single 0.1179s 0.1178s 8.4868 Ops/s 8.4440 Ops/s $\color{#35bf28}+0.51\%$
test_sync 0.1047s 0.1041s 9.6042 Ops/s 9.8500 Ops/s $\color{#d91a1a}-2.50\%$
test_async 0.1992s 95.6411ms 10.4558 Ops/s 12.5393 Ops/s $\textbf{\color{#d91a1a}-16.62\%}$
test_single_pixels 0.1310s 0.1302s 7.6801 Ops/s 7.8072 Ops/s $\color{#d91a1a}-1.63\%$
test_sync_pixels 85.4795ms 83.5288ms 11.9719 Ops/s 12.2773 Ops/s $\color{#d91a1a}-2.49\%$
test_async_pixels 0.1585s 69.3115ms 14.4276 Ops/s 14.7521 Ops/s $\color{#d91a1a}-2.20\%$
test_simple 0.8906s 0.8257s 1.2112 Ops/s 1.2308 Ops/s $\color{#d91a1a}-1.60\%$
test_transformed 1.1360s 1.0764s 0.9290 Ops/s 0.9249 Ops/s $\color{#35bf28}+0.44\%$
test_serial 2.5566s 2.4972s 0.4005 Ops/s 0.3909 Ops/s $\color{#35bf28}+2.45\%$
test_parallel 2.4097s 2.3601s 0.4237 Ops/s 0.4260 Ops/s $\color{#d91a1a}-0.53\%$
test_step_mdp_speed[True-True-True-True-True] 0.1064ms 33.6083μs 29.7546 KOps/s 29.5222 KOps/s $\color{#35bf28}+0.79\%$
test_step_mdp_speed[True-True-True-True-False] 44.3210μs 19.9945μs 50.0138 KOps/s 49.5547 KOps/s $\color{#35bf28}+0.93\%$
test_step_mdp_speed[True-True-True-False-True] 33.8310μs 19.1914μs 52.1067 KOps/s 51.4630 KOps/s $\color{#35bf28}+1.25\%$
test_step_mdp_speed[True-True-True-False-False] 50.7000μs 11.3723μs 87.9330 KOps/s 87.4764 KOps/s $\color{#35bf28}+0.52\%$
test_step_mdp_speed[True-True-False-True-True] 52.1900μs 35.3704μs 28.2722 KOps/s 28.1539 KOps/s $\color{#35bf28}+0.42\%$
test_step_mdp_speed[True-True-False-True-False] 45.8300μs 21.5652μs 46.3711 KOps/s 45.9832 KOps/s $\color{#35bf28}+0.84\%$
test_step_mdp_speed[True-True-False-False-True] 37.2000μs 20.6646μs 48.3920 KOps/s 47.2974 KOps/s $\color{#35bf28}+2.31\%$
test_step_mdp_speed[True-True-False-False-False] 37.5500μs 13.1395μs 76.1067 KOps/s 74.9943 KOps/s $\color{#35bf28}+1.48\%$
test_step_mdp_speed[True-False-True-True-True] 52.3910μs 37.4689μs 26.6888 KOps/s 26.4964 KOps/s $\color{#35bf28}+0.73\%$
test_step_mdp_speed[True-False-True-True-False] 44.4810μs 23.8384μs 41.9491 KOps/s 42.5656 KOps/s $\color{#d91a1a}-1.45\%$
test_step_mdp_speed[True-False-True-False-True] 44.1710μs 20.9866μs 47.6494 KOps/s 46.6632 KOps/s $\color{#35bf28}+2.11\%$
test_step_mdp_speed[True-False-True-False-False] 27.4310μs 13.2565μs 75.4344 KOps/s 75.2413 KOps/s $\color{#35bf28}+0.26\%$
test_step_mdp_speed[True-False-False-True-True] 64.9400μs 38.9456μs 25.6768 KOps/s 25.2240 KOps/s $\color{#35bf28}+1.80\%$
test_step_mdp_speed[True-False-False-True-False] 56.9200μs 25.5255μs 39.1765 KOps/s 38.8698 KOps/s $\color{#35bf28}+0.79\%$
test_step_mdp_speed[True-False-False-False-True] 39.0300μs 22.4513μs 44.5408 KOps/s 43.8090 KOps/s $\color{#35bf28}+1.67\%$
test_step_mdp_speed[True-False-False-False-False] 30.5700μs 14.9831μs 66.7420 KOps/s 65.2995 KOps/s $\color{#35bf28}+2.21\%$
test_step_mdp_speed[False-True-True-True-True] 69.5710μs 37.5573μs 26.6260 KOps/s 26.8461 KOps/s $\color{#d91a1a}-0.82\%$
test_step_mdp_speed[False-True-True-True-False] 43.1600μs 23.8011μs 42.0149 KOps/s 42.2044 KOps/s $\color{#d91a1a}-0.45\%$
test_step_mdp_speed[False-True-True-False-True] 49.9010μs 24.6767μs 40.5241 KOps/s 40.1187 KOps/s $\color{#35bf28}+1.01\%$
test_step_mdp_speed[False-True-True-False-False] 41.1600μs 15.1642μs 65.9447 KOps/s 65.5214 KOps/s $\color{#35bf28}+0.65\%$
test_step_mdp_speed[False-True-False-True-True] 0.1015ms 39.1472μs 25.5446 KOps/s 25.1244 KOps/s $\color{#35bf28}+1.67\%$
test_step_mdp_speed[False-True-False-True-False] 42.1610μs 25.4562μs 39.2832 KOps/s 38.9438 KOps/s $\color{#35bf28}+0.87\%$
test_step_mdp_speed[False-True-False-False-True] 49.7510μs 26.4684μs 37.7809 KOps/s 36.9099 KOps/s $\color{#35bf28}+2.36\%$
test_step_mdp_speed[False-True-False-False-False] 37.9010μs 16.9749μs 58.9106 KOps/s 58.2598 KOps/s $\color{#35bf28}+1.12\%$
test_step_mdp_speed[False-False-True-True-True] 56.9610μs 40.7107μs 24.5636 KOps/s 24.1540 KOps/s $\color{#35bf28}+1.70\%$
test_step_mdp_speed[False-False-True-True-False] 54.1000μs 27.4740μs 36.3981 KOps/s 36.2421 KOps/s $\color{#35bf28}+0.43\%$
test_step_mdp_speed[False-False-True-False-True] 46.9300μs 26.2992μs 38.0239 KOps/s 36.7131 KOps/s $\color{#35bf28}+3.57\%$
test_step_mdp_speed[False-False-True-False-False] 33.9090μs 16.9885μs 58.8635 KOps/s 58.2121 KOps/s $\color{#35bf28}+1.12\%$
test_step_mdp_speed[False-False-False-True-True] 67.2610μs 43.2926μs 23.0986 KOps/s 22.8163 KOps/s $\color{#35bf28}+1.24\%$
test_step_mdp_speed[False-False-False-True-False] 59.8810μs 29.3420μs 34.0809 KOps/s 33.6729 KOps/s $\color{#35bf28}+1.21\%$
test_step_mdp_speed[False-False-False-False-True] 49.3010μs 28.2379μs 35.4134 KOps/s 34.7919 KOps/s $\color{#35bf28}+1.79\%$
test_step_mdp_speed[False-False-False-False-False] 35.8400μs 18.6966μs 53.4858 KOps/s 52.6599 KOps/s $\color{#35bf28}+1.57\%$
test_values[generalized_advantage_estimate-True-True] 26.1089ms 25.6078ms 39.0506 Ops/s 39.3834 Ops/s $\color{#d91a1a}-0.84\%$
test_values[vec_generalized_advantage_estimate-True-True] 80.6870ms 3.1884ms 313.6373 Ops/s 312.7701 Ops/s $\color{#35bf28}+0.28\%$
test_values[td0_return_estimate-False-False] 94.0920μs 65.9436μs 15.1645 KOps/s 14.9929 KOps/s $\color{#35bf28}+1.14\%$
test_values[td1_return_estimate-False-False] 56.8090ms 54.6766ms 18.2894 Ops/s 18.5136 Ops/s $\color{#d91a1a}-1.21\%$
test_values[vec_td1_return_estimate-False-False] 2.1180ms 1.7775ms 562.5801 Ops/s 563.7786 Ops/s $\color{#d91a1a}-0.21\%$
test_values[td_lambda_return_estimate-True-False] 87.9574ms 86.5816ms 11.5498 Ops/s 11.6292 Ops/s $\color{#d91a1a}-0.68\%$
test_values[vec_td_lambda_return_estimate-True-False] 2.1158ms 1.7710ms 564.6550 Ops/s 564.9146 Ops/s $\color{#d91a1a}-0.05\%$
test_gae_speed[generalized_advantage_estimate-False-1-512] 24.5267ms 24.3788ms 41.0193 Ops/s 41.1884 Ops/s $\color{#d91a1a}-0.41\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512] 0.9349ms 0.7080ms 1.4124 KOps/s 1.4091 KOps/s $\color{#35bf28}+0.23\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512] 0.7395ms 0.6581ms 1.5196 KOps/s 1.5160 KOps/s $\color{#35bf28}+0.24\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512] 1.5492ms 1.4612ms 684.3896 Ops/s 683.0210 Ops/s $\color{#35bf28}+0.20\%$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512] 0.9669ms 0.6750ms 1.4815 KOps/s 1.4777 KOps/s $\color{#35bf28}+0.26\%$
test_dqn_speed 7.8160ms 1.4442ms 692.4470 Ops/s 623.6219 Ops/s $\textbf{\color{#35bf28}+11.04\%}$
test_ddpg_speed 3.4520ms 2.9766ms 335.9500 Ops/s 336.8200 Ops/s $\color{#d91a1a}-0.26\%$
test_sac_speed 8.7930ms 8.4332ms 118.5788 Ops/s 117.2806 Ops/s $\color{#35bf28}+1.11\%$
test_redq_speed 11.5995ms 10.5388ms 94.8877 Ops/s 92.8709 Ops/s $\color{#35bf28}+2.17\%$
test_redq_deprec_speed 11.7180ms 11.2859ms 88.6060 Ops/s 85.6436 Ops/s $\color{#35bf28}+3.46\%$
test_td3_speed 8.4797ms 8.3593ms 119.6266 Ops/s 118.3413 Ops/s $\color{#35bf28}+1.09\%$
test_cql_speed 26.3975ms 25.6226ms 39.0280 Ops/s 37.9296 Ops/s $\color{#35bf28}+2.90\%$
test_a2c_speed 5.9014ms 5.5173ms 181.2480 Ops/s 171.9044 Ops/s $\textbf{\color{#35bf28}+5.44\%}$
test_ppo_speed 6.1989ms 5.8517ms 170.8900 Ops/s 163.7351 Ops/s $\color{#35bf28}+4.37\%$
test_reinforce_speed 4.7238ms 4.5220ms 221.1424 Ops/s 213.7713 Ops/s $\color{#35bf28}+3.45\%$
test_iql_speed 20.4907ms 19.5497ms 51.1517 Ops/s 50.9093 Ops/s $\color{#35bf28}+0.48\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 4.7063ms 4.6195ms 216.4739 Ops/s 215.0769 Ops/s $\color{#35bf28}+0.65\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.7055ms 0.5911ms 1.6916 KOps/s 1.6678 KOps/s $\color{#35bf28}+1.43\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 4.1225ms 0.5679ms 1.7608 KOps/s 1.7457 KOps/s $\color{#35bf28}+0.87\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 4.8061ms 4.5868ms 218.0158 Ops/s 216.5617 Ops/s $\color{#35bf28}+0.67\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 0.7110ms 0.5831ms 1.7150 KOps/s 1.6889 KOps/s $\color{#35bf28}+1.55\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.1077s 0.6709ms 1.4906 KOps/s 1.7571 KOps/s $\textbf{\color{#d91a1a}-15.16\%}$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000] 1.6992ms 1.5487ms 645.6859 Ops/s 640.0246 Ops/s $\color{#35bf28}+0.88\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000] 1.6093ms 1.4709ms 679.8540 Ops/s 665.6874 Ops/s $\color{#35bf28}+2.13\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 4.8647ms 4.7762ms 209.3708 Ops/s 209.7059 Ops/s $\color{#d91a1a}-0.16\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 1.3322ms 0.7187ms 1.3915 KOps/s 1.3782 KOps/s $\color{#35bf28}+0.96\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.8796ms 0.6927ms 1.4437 KOps/s 1.4234 KOps/s $\color{#35bf28}+1.42\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 4.8556ms 4.6372ms 215.6492 Ops/s 215.6010 Ops/s $\color{#35bf28}+0.02\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.7063ms 0.5914ms 1.6909 KOps/s 1.6660 KOps/s $\color{#35bf28}+1.49\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 4.3441ms 0.5730ms 1.7453 KOps/s 1.7373 KOps/s $\color{#35bf28}+0.46\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 4.8183ms 4.6192ms 216.4872 Ops/s 216.3011 Ops/s $\color{#35bf28}+0.09\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 0.7311ms 0.5847ms 1.7102 KOps/s 1.6933 KOps/s $\color{#35bf28}+1.00\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.7541ms 0.5624ms 1.7780 KOps/s 1.7394 KOps/s $\color{#35bf28}+2.22\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 4.8882ms 4.7938ms 208.6029 Ops/s 209.2627 Ops/s $\color{#d91a1a}-0.32\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 0.9391ms 0.7218ms 1.3854 KOps/s 1.3788 KOps/s $\color{#35bf28}+0.48\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 4.5088ms 0.6973ms 1.4341 KOps/s 1.4247 KOps/s $\color{#35bf28}+0.66\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 0.1239s 7.2708ms 137.5371 Ops/s 108.9674 Ops/s $\textbf{\color{#35bf28}+26.22\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] 18.1468ms 15.7427ms 63.5214 Ops/s 63.6535 Ops/s $\color{#d91a1a}-0.21\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] 2.3376ms 1.3108ms 762.9005 Ops/s 779.1749 Ops/s $\color{#d91a1a}-2.09\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] 0.1110s 7.0215ms 142.4203 Ops/s 143.9276 Ops/s $\color{#d91a1a}-1.05\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] 0.1156s 17.7567ms 56.3168 Ops/s 56.4456 Ops/s $\color{#d91a1a}-0.23\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] 2.3881ms 1.3620ms 734.2140 Ops/s 767.0740 Ops/s $\color{#d91a1a}-4.28\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] 0.1059s 7.2544ms 137.8473 Ops/s 137.2434 Ops/s $\color{#35bf28}+0.44\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] 18.4075ms 15.9412ms 62.7304 Ops/s 62.8992 Ops/s $\color{#d91a1a}-0.27\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] 2.7364ms 1.6995ms 588.4023 Ops/s 546.4564 Ops/s $\textbf{\color{#35bf28}+7.68\%}$

@vmoens vmoens changed the title [WIP] Dynamic specs [Feature] Dynamic specs May 26, 2024
@vmoens vmoens added the enhancement New feature or request label May 26, 2024
@dtsaras
Copy link

dtsaras commented May 27, 2024

I will start testing this today and provide you with some feedback!

@vmoens vmoens merged commit 8934ed0 into main May 31, 2024
49 of 56 checks passed
@vmoens vmoens deleted the spec-with-neg-shape branch May 31, 2024 13:30
@vmoens vmoens mentioned this pull request Jun 7, 2024
Closed
1 task
@vmoens vmoens mentioned this pull request Aug 27, 2024
Merged
10 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@vmoens @dtsaras @facebook-github-bot