Skip to content

[Feature] LLMEnv and DataLoadingPrimer #2818

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 9 commits into from
Mar 3, 2025
Merged

[Feature] LLMEnv and DataLoadingPrimer #2818

merged 9 commits into from
Mar 3, 2025

Conversation

vmoens
Copy link
Contributor

@vmoens vmoens commented Feb 28, 2025 •
edited
Loading

@vmoens
Update
920a824 
[ghstack-poisoned]
@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Feb 28, 2025 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/2818

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 2 Unrelated Failures

As of commit c062741 with merge base 8c9dc05 (image):

NEW FAILURE - The following job has failed:

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

This was referenced Feb 3, 2025
Merged
Merged
Merged
Merged
Merged
@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Feb 28, 2025
@vmoens
Update
d4ab1eb 
[ghstack-poisoned]
@vmoens vmoens added the enhancement New feature or request label Feb 28, 2025
@vmoens
Copy link
Contributor Author

vmoens commented Feb 28, 2025

cc @mikaylagawarecki

@vmoens
Update
ec1c98d 
[ghstack-poisoned]
@vmoens
Update
be6c379 
[ghstack-poisoned]
@vmoens
Update
ef09077 
[ghstack-poisoned]
@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 28, 2025 •
edited
Loading

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of CPU Benchmark Tests

Total Benchmarks: 149. Improved: $\large\color{#35bf28}31$. Worsened: $\large\color{#d91a1a}17$.

Expand to view detailed results
Name Max Mean Ops Ops on Repo HEAD Change
test_simple 0.6098s 0.5216s 1.9173 Ops/s 1.8678 Ops/s $\color{#35bf28}+2.65\%$
test_transformed 1.0936s 0.9970s 1.0030 Ops/s 0.9051 Ops/s $\textbf{\color{#35bf28}+10.82\%}$
test_serial 1.6328s 1.5188s 0.6584 Ops/s 0.6061 Ops/s $\textbf{\color{#35bf28}+8.64\%}$
test_parallel 1.4364s 1.3266s 0.7538 Ops/s 0.6964 Ops/s $\textbf{\color{#35bf28}+8.25\%}$
test_step_mdp_speed[True-True-True-True-True] 0.2357ms 30.4047μs 32.8897 KOps/s 32.4999 KOps/s $\color{#35bf28}+1.20\%$
test_step_mdp_speed[True-True-True-True-False] 57.4870μs 18.4073μs 54.3262 KOps/s 52.2724 KOps/s $\color{#35bf28}+3.93\%$
test_step_mdp_speed[True-True-True-False-True] 73.9680μs 17.3534μs 57.6255 KOps/s 57.2185 KOps/s $\color{#35bf28}+0.71\%$
test_step_mdp_speed[True-True-True-False-False] 43.0910μs 10.5585μs 94.7103 KOps/s 90.4476 KOps/s $\color{#35bf28}+4.71\%$
test_step_mdp_speed[True-True-False-True-True] 91.6710μs 32.8144μs 30.4744 KOps/s 30.6735 KOps/s $\color{#d91a1a}-0.65\%$
test_step_mdp_speed[True-True-False-True-False] 66.5040μs 20.2258μs 49.4419 KOps/s 48.3641 KOps/s $\color{#35bf28}+2.23\%$
test_step_mdp_speed[True-True-False-False-True] 52.0670μs 19.6494μs 50.8920 KOps/s 51.8435 KOps/s $\color{#d91a1a}-1.84\%$
test_step_mdp_speed[True-True-False-False-False] 58.9990μs 12.4013μs 80.6365 KOps/s 81.4040 KOps/s $\color{#d91a1a}-0.94\%$
test_step_mdp_speed[True-False-True-True-True] 71.8650μs 35.4904μs 28.1767 KOps/s 28.1459 KOps/s $\color{#35bf28}+0.11\%$
test_step_mdp_speed[True-False-True-True-False] 84.3510μs 22.1265μs 45.1947 KOps/s 43.9596 KOps/s $\color{#35bf28}+2.81\%$
test_step_mdp_speed[True-False-True-False-True] 72.4530μs 19.6002μs 51.0199 KOps/s 50.8449 KOps/s $\color{#35bf28}+0.34\%$
test_step_mdp_speed[True-False-True-False-False] 40.1750μs 12.4221μs 80.5016 KOps/s 79.5342 KOps/s $\color{#35bf28}+1.22\%$
test_step_mdp_speed[True-False-False-True-True] 0.1175ms 36.2006μs 27.6238 KOps/s 26.9434 KOps/s $\color{#35bf28}+2.53\%$
test_step_mdp_speed[True-False-False-True-False] 89.9880μs 23.9793μs 41.7026 KOps/s 40.9470 KOps/s $\color{#35bf28}+1.85\%$
test_step_mdp_speed[True-False-False-False-True] 53.4400μs 21.4847μs 46.5447 KOps/s 45.2258 KOps/s $\color{#35bf28}+2.92\%$
test_step_mdp_speed[True-False-False-False-False] 72.0340μs 14.2106μs 70.3702 KOps/s 70.4708 KOps/s $\color{#d91a1a}-0.14\%$
test_step_mdp_speed[False-True-True-True-True] 70.9630μs 34.8895μs 28.6619 KOps/s 27.5202 KOps/s $\color{#35bf28}+4.15\%$
test_step_mdp_speed[False-True-True-True-False] 0.6424ms 22.0684μs 45.3138 KOps/s 44.5156 KOps/s $\color{#35bf28}+1.79\%$
test_step_mdp_speed[False-True-True-False-True] 56.1550μs 22.2560μs 44.9317 KOps/s 44.6066 KOps/s $\color{#35bf28}+0.73\%$
test_step_mdp_speed[False-True-True-False-False] 66.4340μs 13.8042μs 72.4419 KOps/s 72.4334 KOps/s $\color{#35bf28}+0.01\%$
test_step_mdp_speed[False-True-False-True-True] 91.9510μs 36.4661μs 27.4227 KOps/s 27.1171 KOps/s $\color{#35bf28}+1.13\%$
test_step_mdp_speed[False-True-False-True-False] 53.0500μs 23.8676μs 41.8979 KOps/s 40.4962 KOps/s $\color{#35bf28}+3.46\%$
test_step_mdp_speed[False-True-False-False-True] 3.6891ms 23.9699μs 41.7189 KOps/s 41.3473 KOps/s $\color{#35bf28}+0.90\%$
test_step_mdp_speed[False-True-False-False-False] 45.5450μs 15.6213μs 64.0154 KOps/s 62.6428 KOps/s $\color{#35bf28}+2.19\%$
test_step_mdp_speed[False-False-True-True-True] 95.3580μs 38.4673μs 25.9961 KOps/s 25.7183 KOps/s $\color{#35bf28}+1.08\%$
test_step_mdp_speed[False-False-True-True-False] 83.5660μs 25.7584μs 38.8223 KOps/s 38.4633 KOps/s $\color{#35bf28}+0.93\%$
test_step_mdp_speed[False-False-True-False-True] 60.6930μs 24.5061μs 40.8062 KOps/s 41.1998 KOps/s $\color{#d91a1a}-0.96\%$
test_step_mdp_speed[False-False-True-False-False] 63.4180μs 15.6084μs 64.0679 KOps/s 64.7553 KOps/s $\color{#d91a1a}-1.06\%$
test_step_mdp_speed[False-False-False-True-True] 96.2390μs 39.9206μs 25.0497 KOps/s 24.5582 KOps/s $\color{#35bf28}+2.00\%$
test_step_mdp_speed[False-False-False-True-False] 56.1740μs 27.1181μs 36.8758 KOps/s 36.6335 KOps/s $\color{#35bf28}+0.66\%$
test_step_mdp_speed[False-False-False-False-True] 75.3210μs 25.5177μs 39.1885 KOps/s 38.8241 KOps/s $\color{#35bf28}+0.94\%$
test_step_mdp_speed[False-False-False-False-False] 50.1430μs 17.1252μs 58.3933 KOps/s 59.0143 KOps/s $\color{#d91a1a}-1.05\%$
test_values[generalized_advantage_estimate-True-True] 10.7540ms 9.7740ms 102.3128 Ops/s 99.1852 Ops/s $\color{#35bf28}+3.15\%$
test_values[vec_generalized_advantage_estimate-True-True] 30.0287ms 26.3872ms 37.8972 Ops/s 30.6808 Ops/s $\textbf{\color{#35bf28}+23.52\%}$
test_values[td0_return_estimate-False-False] 0.2673ms 0.1783ms 5.6076 KOps/s 5.3756 KOps/s $\color{#35bf28}+4.32\%$
test_values[td1_return_estimate-False-False] 25.2693ms 24.7164ms 40.4589 Ops/s 40.1714 Ops/s $\color{#35bf28}+0.72\%$
test_values[vec_td1_return_estimate-False-False] 29.6887ms 26.1504ms 38.2403 Ops/s 31.0029 Ops/s $\textbf{\color{#35bf28}+23.34\%}$
test_values[td_lambda_return_estimate-True-False] 37.8879ms 35.2004ms 28.4088 Ops/s 27.6934 Ops/s $\color{#35bf28}+2.58\%$
test_values[vec_td_lambda_return_estimate-True-False] 29.9753ms 26.9066ms 37.1656 Ops/s 31.7271 Ops/s $\textbf{\color{#35bf28}+17.14\%}$
test_gae_speed[generalized_advantage_estimate-False-1-512] 9.5009ms 8.5969ms 116.3210 Ops/s 114.2503 Ops/s $\color{#35bf28}+1.81\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512] 2.4743ms 1.9442ms 514.3538 Ops/s 497.1754 Ops/s $\color{#35bf28}+3.46\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512] 0.5056ms 0.3870ms 2.5839 KOps/s 2.4531 KOps/s $\textbf{\color{#35bf28}+5.33\%}$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512] 49.1042ms 42.3428ms 23.6168 Ops/s 18.4140 Ops/s $\textbf{\color{#35bf28}+28.25\%}$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512] 3.5816ms 3.4297ms 291.5694 Ops/s 267.5086 Ops/s $\textbf{\color{#35bf28}+8.99\%}$
test_dqn_speed[False-None] 5.9167ms 1.4382ms 695.3334 Ops/s 655.5073 Ops/s $\textbf{\color{#35bf28}+6.08\%}$
test_dqn_speed[False-backward] 2.1310ms 1.9472ms 513.5691 Ops/s 485.7237 Ops/s $\textbf{\color{#35bf28}+5.73\%}$
test_dqn_speed[True-None] 0.9369ms 0.4953ms 2.0191 KOps/s 1.9122 KOps/s $\textbf{\color{#35bf28}+5.59\%}$
test_dqn_speed[True-backward] 1.0718ms 1.0109ms 989.1831 Ops/s 991.8202 Ops/s $\color{#d91a1a}-0.27\%$
test_dqn_speed[reduce-overhead-None] 0.7587ms 0.4873ms 2.0523 KOps/s 1.8798 KOps/s $\textbf{\color{#35bf28}+9.18\%}$
test_dqn_speed[reduce-overhead-backward] 1.3608ms 1.0200ms 980.3743 Ops/s 912.7944 Ops/s $\textbf{\color{#35bf28}+7.40\%}$
test_ddpg_speed[False-None] 3.7284ms 2.9617ms 337.6412 Ops/s 317.4784 Ops/s $\textbf{\color{#35bf28}+6.35\%}$
test_ddpg_speed[False-backward] 4.1686ms 4.0214ms 248.6684 Ops/s 228.7713 Ops/s $\textbf{\color{#35bf28}+8.70\%}$
test_ddpg_speed[True-None] 1.5980ms 1.2596ms 793.9046 Ops/s 746.7067 Ops/s $\textbf{\color{#35bf28}+6.32\%}$
test_ddpg_speed[True-backward] 2.4860ms 2.2425ms 445.9223 Ops/s 394.5842 Ops/s $\textbf{\color{#35bf28}+13.01\%}$
test_ddpg_speed[reduce-overhead-None] 1.6229ms 1.2746ms 784.5460 Ops/s 734.3118 Ops/s $\textbf{\color{#35bf28}+6.84\%}$
test_ddpg_speed[reduce-overhead-backward] 2.4214ms 2.2456ms 445.3122 Ops/s 423.9911 Ops/s $\textbf{\color{#35bf28}+5.03\%}$
test_sac_speed[False-None] 10.3218ms 8.1897ms 122.1053 Ops/s 115.1070 Ops/s $\textbf{\color{#35bf28}+6.08\%}$
test_sac_speed[False-backward] 13.9086ms 11.1159ms 89.9609 Ops/s 86.5356 Ops/s $\color{#35bf28}+3.96\%$
test_sac_speed[True-None] 3.1756ms 2.1586ms 463.2586 Ops/s 445.0305 Ops/s $\color{#35bf28}+4.10\%$
test_sac_speed[True-backward] 4.1770ms 3.8501ms 259.7334 Ops/s 256.1423 Ops/s $\color{#35bf28}+1.40\%$
test_sac_speed[reduce-overhead-None] 2.4400ms 2.1294ms 469.6268 Ops/s 425.6752 Ops/s $\textbf{\color{#35bf28}+10.33\%}$
test_sac_speed[reduce-overhead-backward] 4.1230ms 3.8580ms 259.1990 Ops/s 243.9604 Ops/s $\textbf{\color{#35bf28}+6.25\%}$
test_redq_speed[False-None] 15.6117ms 13.5496ms 73.8027 Ops/s 68.6369 Ops/s $\textbf{\color{#35bf28}+7.53\%}$
test_redq_speed[False-backward] 26.8648ms 23.0900ms 43.3087 Ops/s 36.2317 Ops/s $\textbf{\color{#35bf28}+19.53\%}$
test_redq_speed[True-None] 6.9274ms 4.9921ms 200.3176 Ops/s 186.9270 Ops/s $\textbf{\color{#35bf28}+7.16\%}$
test_redq_speed[True-backward] 16.0083ms 13.1813ms 75.8652 Ops/s 72.7202 Ops/s $\color{#35bf28}+4.32\%$
test_redq_speed[reduce-overhead-None] 5.8469ms 5.0284ms 198.8701 Ops/s 178.6074 Ops/s $\textbf{\color{#35bf28}+11.34\%}$
test_redq_speed[reduce-overhead-backward] 15.0761ms 12.6681ms 78.9382 Ops/s 71.4366 Ops/s $\textbf{\color{#35bf28}+10.50\%}$
test_redq_deprec_speed[False-None] 14.7599ms 13.0965ms 76.3564 Ops/s 72.8401 Ops/s $\color{#35bf28}+4.83\%$
test_redq_deprec_speed[False-backward] 20.3712ms 18.7473ms 53.3410 Ops/s 50.2552 Ops/s $\textbf{\color{#35bf28}+6.14\%}$
test_redq_deprec_speed[True-None] 4.7365ms 3.8791ms 257.7914 Ops/s 248.2100 Ops/s $\color{#35bf28}+3.86\%$
test_redq_deprec_speed[True-backward] 9.1588ms 8.3232ms 120.1456 Ops/s 115.1785 Ops/s $\color{#35bf28}+4.31\%$
test_redq_deprec_speed[reduce-overhead-None] 4.8860ms 4.0094ms 249.4156 Ops/s 248.5660 Ops/s $\color{#35bf28}+0.34\%$
test_redq_deprec_speed[reduce-overhead-backward] 13.1522ms 9.1238ms 109.6029 Ops/s 116.3600 Ops/s $\textbf{\color{#d91a1a}-5.81\%}$
test_td3_speed[False-None] 8.8623ms 8.2256ms 121.5711 Ops/s 119.8303 Ops/s $\color{#35bf28}+1.45\%$
test_td3_speed[False-backward] 14.0571ms 11.1863ms 89.3952 Ops/s 93.6157 Ops/s $\color{#d91a1a}-4.51\%$
test_td3_speed[True-None] 2.0299ms 1.8527ms 539.7385 Ops/s 534.2558 Ops/s $\color{#35bf28}+1.03\%$
test_td3_speed[True-backward] 4.0927ms 3.5173ms 284.3120 Ops/s 282.3561 Ops/s $\color{#35bf28}+0.69\%$
test_td3_speed[reduce-overhead-None] 2.0809ms 1.8253ms 547.8569 Ops/s 539.2487 Ops/s $\color{#35bf28}+1.60\%$
test_td3_speed[reduce-overhead-backward] 3.8717ms 3.4491ms 289.9326 Ops/s 279.9627 Ops/s $\color{#35bf28}+3.56\%$
test_cql_speed[False-None] 43.0657ms 38.7682ms 25.7943 Ops/s 26.0421 Ops/s $\color{#d91a1a}-0.95\%$
test_cql_speed[False-backward] 50.2080ms 48.2732ms 20.7154 Ops/s 19.7340 Ops/s $\color{#35bf28}+4.97\%$
test_cql_speed[True-None] 17.6958ms 16.4837ms 60.6660 Ops/s 58.2822 Ops/s $\color{#35bf28}+4.09\%$
test_cql_speed[True-backward] 26.4329ms 23.5986ms 42.3754 Ops/s 38.7468 Ops/s $\textbf{\color{#35bf28}+9.36\%}$
test_cql_speed[reduce-overhead-None] 19.3447ms 17.5566ms 56.9585 Ops/s 56.0857 Ops/s $\color{#35bf28}+1.56\%$
test_cql_speed[reduce-overhead-backward] 29.7136ms 26.1486ms 38.2429 Ops/s 41.0025 Ops/s $\textbf{\color{#d91a1a}-6.73\%}$
test_a2c_speed[False-None] 9.2623ms 7.5393ms 132.6380 Ops/s 135.3819 Ops/s $\color{#d91a1a}-2.03\%$
test_a2c_speed[False-backward] 17.1642ms 15.1859ms 65.8505 Ops/s 68.0403 Ops/s $\color{#d91a1a}-3.22\%$
test_a2c_speed[True-None] 4.5646ms 3.9629ms 252.3404 Ops/s 261.3532 Ops/s $\color{#d91a1a}-3.45\%$
test_a2c_speed[True-backward] 11.4379ms 10.8295ms 92.3404 Ops/s 95.8267 Ops/s $\color{#d91a1a}-3.64\%$
test_a2c_speed[reduce-overhead-None] 4.1423ms 3.8855ms 257.3689 Ops/s 262.5327 Ops/s $\color{#d91a1a}-1.97\%$
test_a2c_speed[reduce-overhead-backward] 12.2499ms 10.7079ms 93.3894 Ops/s 95.4273 Ops/s $\color{#d91a1a}-2.14\%$
test_ppo_speed[False-None] 9.3634ms 7.9458ms 125.8531 Ops/s 129.5358 Ops/s $\color{#d91a1a}-2.84\%$
test_ppo_speed[False-backward] 18.5367ms 16.2311ms 61.6100 Ops/s 65.7828 Ops/s $\textbf{\color{#d91a1a}-6.34\%}$
test_ppo_speed[True-None] 5.0701ms 4.3451ms 230.1423 Ops/s 237.5353 Ops/s $\color{#d91a1a}-3.11\%$
test_ppo_speed[True-backward] 12.5491ms 10.8517ms 92.1516 Ops/s 97.0141 Ops/s $\textbf{\color{#d91a1a}-5.01\%}$
test_ppo_speed[reduce-overhead-None] 4.8276ms 4.3598ms 229.3688 Ops/s 239.1547 Ops/s $\color{#d91a1a}-4.09\%$
test_ppo_speed[reduce-overhead-backward] 11.6682ms 10.6081ms 94.2672 Ops/s 96.6738 Ops/s $\color{#d91a1a}-2.49\%$
test_reinforce_speed[False-None] 7.9077ms 6.9566ms 143.7489 Ops/s 146.5984 Ops/s $\color{#d91a1a}-1.94\%$
test_reinforce_speed[False-backward] 11.8293ms 10.4970ms 95.2655 Ops/s 93.1552 Ops/s $\color{#35bf28}+2.27\%$
test_reinforce_speed[True-None] 4.2076ms 3.3194ms 301.2635 Ops/s 305.9909 Ops/s $\color{#d91a1a}-1.54\%$
test_reinforce_speed[True-backward] 10.0574ms 9.4852ms 105.4269 Ops/s 107.8838 Ops/s $\color{#d91a1a}-2.28\%$
test_reinforce_speed[reduce-overhead-None] 3.7745ms 3.3072ms 302.3740 Ops/s 314.5462 Ops/s $\color{#d91a1a}-3.87\%$
test_reinforce_speed[reduce-overhead-backward] 11.0097ms 9.5233ms 105.0052 Ops/s 109.4692 Ops/s $\color{#d91a1a}-4.08\%$
test_iql_speed[False-None] 38.0495ms 35.5070ms 28.1635 Ops/s 30.0037 Ops/s $\textbf{\color{#d91a1a}-6.13\%}$
test_iql_speed[False-backward] 53.7219ms 50.1619ms 19.9354 Ops/s 21.2000 Ops/s $\textbf{\color{#d91a1a}-5.97\%}$
test_iql_speed[True-None] 15.1979ms 12.9776ms 77.0557 Ops/s 85.6197 Ops/s $\textbf{\color{#d91a1a}-10.00\%}$
test_iql_speed[True-backward] 27.6359ms 24.8032ms 40.3175 Ops/s 41.8847 Ops/s $\color{#d91a1a}-3.74\%$
test_iql_speed[reduce-overhead-None] 14.2304ms 12.0609ms 82.9128 Ops/s 83.0658 Ops/s $\color{#d91a1a}-0.18\%$
test_iql_speed[reduce-overhead-backward] 25.8310ms 23.4098ms 42.7171 Ops/s 43.9072 Ops/s $\color{#d91a1a}-2.71\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 7.9382ms 5.1009ms 196.0433 Ops/s 200.3276 Ops/s $\color{#d91a1a}-2.14\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 1.0110ms 0.6129ms 1.6315 KOps/s 1.6904 KOps/s $\color{#d91a1a}-3.48\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.8515ms 0.5545ms 1.8034 KOps/s 1.8284 KOps/s $\color{#d91a1a}-1.37\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 9.8023ms 5.0063ms 199.7492 Ops/s 215.3348 Ops/s $\textbf{\color{#d91a1a}-7.24\%}$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 1.1581ms 0.5999ms 1.6670 KOps/s 1.7852 KOps/s $\textbf{\color{#d91a1a}-6.62\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.9209ms 0.5603ms 1.7848 KOps/s 1.8623 KOps/s $\color{#d91a1a}-4.16\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000] 2.6204ms 1.8600ms 537.6474 Ops/s 546.3898 Ops/s $\color{#d91a1a}-1.60\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000] 2.2878ms 1.7552ms 569.7242 Ops/s 575.1237 Ops/s $\color{#d91a1a}-0.94\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 5.7038ms 4.9425ms 202.3287 Ops/s 208.6102 Ops/s $\color{#d91a1a}-3.01\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 1.3972ms 0.7334ms 1.3635 KOps/s 1.4058 KOps/s $\color{#d91a1a}-3.01\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 1.0837ms 0.7083ms 1.4119 KOps/s 1.4601 KOps/s $\color{#d91a1a}-3.30\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 7.3573ms 4.8274ms 207.1512 Ops/s 214.3800 Ops/s $\color{#d91a1a}-3.37\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 2.6989ms 0.5918ms 1.6899 KOps/s 1.7345 KOps/s $\color{#d91a1a}-2.57\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.9859ms 0.5612ms 1.7819 KOps/s 1.8190 KOps/s $\color{#d91a1a}-2.04\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 5.5770ms 4.7858ms 208.9535 Ops/s 213.3941 Ops/s $\color{#d91a1a}-2.08\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 2.5424ms 0.5676ms 1.7619 KOps/s 1.8281 KOps/s $\color{#d91a1a}-3.62\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.7722ms 0.5462ms 1.8308 KOps/s 1.7922 KOps/s $\color{#35bf28}+2.15\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 7.6915ms 5.0606ms 197.6061 Ops/s 212.0495 Ops/s $\textbf{\color{#d91a1a}-6.81\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 1.3874ms 0.7451ms 1.3421 KOps/s 1.3873 KOps/s $\color{#d91a1a}-3.26\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 1.0043ms 0.6920ms 1.4452 KOps/s 1.4310 KOps/s $\color{#35bf28}+0.99\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 5.9901ms 4.5590ms 219.3441 Ops/s 229.9673 Ops/s $\color{#d91a1a}-4.62\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] 9.8673ms 2.9190ms 342.5859 Ops/s 393.1856 Ops/s $\textbf{\color{#d91a1a}-12.87\%}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] 2.2938ms 1.5510ms 644.7452 Ops/s 681.7693 Ops/s $\textbf{\color{#d91a1a}-5.43\%}$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] 0.5329s 15.3456ms 65.1651 Ops/s 240.4227 Ops/s $\textbf{\color{#d91a1a}-72.90\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] 3.9656ms 2.6276ms 380.5760 Ops/s 369.3642 Ops/s $\color{#35bf28}+3.04\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] 8.3705ms 1.6974ms 589.1251 Ops/s 681.3982 Ops/s $\textbf{\color{#d91a1a}-13.54\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] 6.9923ms 4.7106ms 212.2887 Ops/s 30.3654 Ops/s $\textbf{\color{#35bf28}+599.11\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] 8.4292ms 2.8801ms 347.2120 Ops/s 387.0746 Ops/s $\textbf{\color{#d91a1a}-10.30\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] 5.4938ms 1.7099ms 584.8388 Ops/s 637.3393 Ops/s $\textbf{\color{#d91a1a}-8.24\%}$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-True] 13.0660ms 12.4368ms 80.4065 Ops/s 83.0067 Ops/s $\color{#d91a1a}-3.13\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-False] 20.8102ms 15.5615ms 64.2610 Ops/s 68.1080 Ops/s $\textbf{\color{#d91a1a}-5.65\%}$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-True] 22.2379ms 21.5399ms 46.4254 Ops/s 47.8617 Ops/s $\color{#d91a1a}-3.00\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-False] 16.0883ms 15.0844ms 66.2935 Ops/s 66.5338 Ops/s $\color{#d91a1a}-0.36\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-True] 24.8060ms 21.6524ms 46.1842 Ops/s 48.0478 Ops/s $\color{#d91a1a}-3.88\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-False] 18.8105ms 16.5879ms 60.2848 Ops/s 61.1087 Ops/s $\color{#d91a1a}-1.35\%$

@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 28, 2025 •
edited
Loading

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of GPU Benchmark Tests

Total Benchmarks: 149. Improved: $\large\color{#35bf28}12$. Worsened: $\large\color{#d91a1a}14$.

Expand to view detailed results
Name Max Mean Ops Ops on Repo HEAD Change
test_simple 0.9134s 0.8265s 1.2099 Ops/s 1.2013 Ops/s $\color{#35bf28}+0.71\%$
test_transformed 1.5498s 1.4618s 0.6841 Ops/s 0.6763 Ops/s $\color{#35bf28}+1.15\%$
test_serial 2.4141s 2.3270s 0.4297 Ops/s 0.4288 Ops/s $\color{#35bf28}+0.23\%$
test_parallel 1.9620s 1.8836s 0.5309 Ops/s 0.5347 Ops/s $\color{#d91a1a}-0.71\%$
test_step_mdp_speed[True-True-True-True-True] 0.1239ms 38.4858μs 25.9836 KOps/s 25.3761 KOps/s $\color{#35bf28}+2.39\%$
test_step_mdp_speed[True-True-True-True-False] 54.0610μs 22.8839μs 43.6988 KOps/s 43.8072 KOps/s $\color{#d91a1a}-0.25\%$
test_step_mdp_speed[True-True-True-False-True] 58.7710μs 21.6008μs 46.2946 KOps/s 46.5309 KOps/s $\color{#d91a1a}-0.51\%$
test_step_mdp_speed[True-True-True-False-False] 39.4600μs 12.8582μs 77.7714 KOps/s 80.3070 KOps/s $\color{#d91a1a}-3.16\%$
test_step_mdp_speed[True-True-False-True-True] 0.1138ms 41.7357μs 23.9603 KOps/s 24.2976 KOps/s $\color{#d91a1a}-1.39\%$
test_step_mdp_speed[True-True-False-True-False] 61.5410μs 24.9266μs 40.1178 KOps/s 40.1367 KOps/s $\color{#d91a1a}-0.05\%$
test_step_mdp_speed[True-True-False-False-True] 0.1139ms 24.1246μs 41.4515 KOps/s 40.7892 KOps/s $\color{#35bf28}+1.62\%$
test_step_mdp_speed[True-True-False-False-False] 43.3410μs 15.0848μs 66.2919 KOps/s 66.6664 KOps/s $\color{#d91a1a}-0.56\%$
test_step_mdp_speed[True-False-True-True-True] 76.2710μs 44.0546μs 22.6991 KOps/s 22.4007 KOps/s $\color{#35bf28}+1.33\%$
test_step_mdp_speed[True-False-True-True-False] 54.0310μs 26.9974μs 37.0406 KOps/s 36.5497 KOps/s $\color{#35bf28}+1.34\%$
test_step_mdp_speed[True-False-True-False-True] 57.0710μs 23.9924μs 41.6798 KOps/s 41.2089 KOps/s $\color{#35bf28}+1.14\%$
test_step_mdp_speed[True-False-True-False-False] 42.3710μs 15.1245μs 66.1177 KOps/s 65.7786 KOps/s $\color{#35bf28}+0.52\%$
test_step_mdp_speed[True-False-False-True-True] 80.6510μs 46.6172μs 21.4513 KOps/s 21.7485 KOps/s $\color{#d91a1a}-1.37\%$
test_step_mdp_speed[True-False-False-True-False] 56.9710μs 30.3085μs 32.9941 KOps/s 33.5728 KOps/s $\color{#d91a1a}-1.72\%$
test_step_mdp_speed[True-False-False-False-True] 59.9310μs 26.6373μs 37.5414 KOps/s 37.3263 KOps/s $\color{#35bf28}+0.58\%$
test_step_mdp_speed[True-False-False-False-False] 51.7810μs 17.3991μs 57.4741 KOps/s 57.2349 KOps/s $\color{#35bf28}+0.42\%$
test_step_mdp_speed[False-True-True-True-True] 76.0910μs 43.0637μs 23.2214 KOps/s 22.1970 KOps/s $\color{#35bf28}+4.61\%$
test_step_mdp_speed[False-True-True-True-False] 55.8310μs 27.0257μs 37.0018 KOps/s 35.4495 KOps/s $\color{#35bf28}+4.38\%$
test_step_mdp_speed[False-True-True-False-True] 64.7410μs 27.6839μs 36.1220 KOps/s 35.4494 KOps/s $\color{#35bf28}+1.90\%$
test_step_mdp_speed[False-True-True-False-False] 48.5810μs 16.5743μs 60.3345 KOps/s 59.0093 KOps/s $\color{#35bf28}+2.25\%$
test_step_mdp_speed[False-True-False-True-True] 92.7720μs 46.0309μs 21.7245 KOps/s 21.9515 KOps/s $\color{#d91a1a}-1.03\%$
test_step_mdp_speed[False-True-False-True-False] 66.2520μs 29.7960μs 33.5616 KOps/s 33.5295 KOps/s $\color{#35bf28}+0.10\%$
test_step_mdp_speed[False-True-False-False-True] 3.2641ms 30.3529μs 32.9458 KOps/s 32.3516 KOps/s $\color{#35bf28}+1.84\%$
test_step_mdp_speed[False-True-False-False-False] 45.6510μs 19.0896μs 52.3845 KOps/s 52.1105 KOps/s $\color{#35bf28}+0.53\%$
test_step_mdp_speed[False-False-True-True-True] 90.6410μs 48.8472μs 20.4720 KOps/s 20.6318 KOps/s $\color{#d91a1a}-0.77\%$
test_step_mdp_speed[False-False-True-True-False] 67.8210μs 32.3758μs 30.8872 KOps/s 30.7700 KOps/s $\color{#35bf28}+0.38\%$
test_step_mdp_speed[False-False-True-False-True] 58.3010μs 30.1793μs 33.1353 KOps/s 31.8144 KOps/s $\color{#35bf28}+4.15\%$
test_step_mdp_speed[False-False-True-False-False] 47.1310μs 18.7894μs 53.2216 KOps/s 52.0189 KOps/s $\color{#35bf28}+2.31\%$
test_step_mdp_speed[False-False-False-True-True] 0.1002ms 50.8420μs 19.6688 KOps/s 19.5599 KOps/s $\color{#35bf28}+0.56\%$
test_step_mdp_speed[False-False-False-True-False] 65.1910μs 34.6043μs 28.8981 KOps/s 28.7282 KOps/s $\color{#35bf28}+0.59\%$
test_step_mdp_speed[False-False-False-False-True] 67.4810μs 32.1839μs 31.0714 KOps/s 31.8089 KOps/s $\color{#d91a1a}-2.32\%$
test_step_mdp_speed[False-False-False-False-False] 57.5510μs 20.7547μs 48.1818 KOps/s 46.9026 KOps/s $\color{#35bf28}+2.73\%$
test_values[generalized_advantage_estimate-True-True] 25.6089ms 24.7402ms 40.4200 Ops/s 41.0489 Ops/s $\color{#d91a1a}-1.53\%$
test_values[vec_generalized_advantage_estimate-True-True] 0.1134s 3.1665ms 315.8059 Ops/s 353.5403 Ops/s $\textbf{\color{#d91a1a}-10.67\%}$
test_values[td0_return_estimate-False-False] 0.1094ms 79.6053μs 12.5620 KOps/s 12.6989 KOps/s $\color{#d91a1a}-1.08\%$
test_values[td1_return_estimate-False-False] 54.8721ms 54.1923ms 18.4528 Ops/s 18.1253 Ops/s $\color{#35bf28}+1.81\%$
test_values[vec_td1_return_estimate-False-False] 1.2554ms 1.0790ms 926.8237 Ops/s 928.5549 Ops/s $\color{#d91a1a}-0.19\%$
test_values[td_lambda_return_estimate-True-False] 90.7821ms 86.8819ms 11.5099 Ops/s 11.7408 Ops/s $\color{#d91a1a}-1.97\%$
test_values[vec_td_lambda_return_estimate-True-False] 1.2071ms 1.0747ms 930.5152 Ops/s 931.0466 Ops/s $\color{#d91a1a}-0.06\%$
test_gae_speed[generalized_advantage_estimate-False-1-512] 25.1454ms 24.4437ms 40.9104 Ops/s 39.1155 Ops/s $\color{#35bf28}+4.59\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512] 1.0124ms 0.7476ms 1.3376 KOps/s 1.3410 KOps/s $\color{#d91a1a}-0.26\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512] 0.8050ms 0.6634ms 1.5073 KOps/s 1.5214 KOps/s $\color{#d91a1a}-0.92\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512] 1.6397ms 1.4854ms 673.2029 Ops/s 676.3684 Ops/s $\color{#d91a1a}-0.47\%$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512] 0.8161ms 0.6973ms 1.4340 KOps/s 1.4371 KOps/s $\color{#d91a1a}-0.22\%$
test_dqn_speed[False-None] 1.5947ms 1.5049ms 664.4996 Ops/s 667.1754 Ops/s $\color{#d91a1a}-0.40\%$
test_dqn_speed[False-backward] 2.3707ms 2.1316ms 469.1361 Ops/s 487.1729 Ops/s $\color{#d91a1a}-3.70\%$
test_dqn_speed[True-None] 0.6895ms 0.5449ms 1.8353 KOps/s 1.8170 KOps/s $\color{#35bf28}+1.01\%$
test_dqn_speed[True-backward] 1.2143ms 1.1339ms 881.8954 Ops/s 894.0750 Ops/s $\color{#d91a1a}-1.36\%$
test_dqn_speed[reduce-overhead-None] 0.7709ms 0.5841ms 1.7120 KOps/s 1.7908 KOps/s $\color{#d91a1a}-4.40\%$
test_dqn_speed[reduce-overhead-backward] 1.0409ms 0.9486ms 1.0542 KOps/s 1.0374 KOps/s $\color{#35bf28}+1.62\%$
test_ddpg_speed[False-None] 3.0990ms 2.8121ms 355.6021 Ops/s 357.0787 Ops/s $\color{#d91a1a}-0.41\%$
test_ddpg_speed[False-backward] 4.5529ms 4.1263ms 242.3453 Ops/s 245.4228 Ops/s $\color{#d91a1a}-1.25\%$
test_ddpg_speed[True-None] 1.4128ms 1.3097ms 763.5166 Ops/s 765.0239 Ops/s $\color{#d91a1a}-0.20\%$
test_ddpg_speed[True-backward] 2.5812ms 2.5316ms 395.0111 Ops/s 390.4923 Ops/s $\color{#35bf28}+1.16\%$
test_ddpg_speed[reduce-overhead-None] 1.3890ms 1.3162ms 759.7395 Ops/s 761.3595 Ops/s $\color{#d91a1a}-0.21\%$
test_ddpg_speed[reduce-overhead-backward] 2.0678ms 2.0070ms 498.2559 Ops/s 498.4498 Ops/s $\color{#d91a1a}-0.04\%$
test_sac_speed[False-None] 8.5214ms 8.0696ms 123.9212 Ops/s 124.1751 Ops/s $\color{#d91a1a}-0.20\%$
test_sac_speed[False-backward] 11.7699ms 11.2494ms 88.8934 Ops/s 88.8124 Ops/s $\color{#35bf28}+0.09\%$
test_sac_speed[True-None] 1.8541ms 1.7917ms 558.1249 Ops/s 529.5923 Ops/s $\textbf{\color{#35bf28}+5.39\%}$
test_sac_speed[True-backward] 3.7878ms 3.7026ms 270.0786 Ops/s 281.7683 Ops/s $\color{#d91a1a}-4.15\%$
test_sac_speed[reduce-overhead-None] 20.4150ms 11.7007ms 85.4653 Ops/s 85.5541 Ops/s $\color{#d91a1a}-0.10\%$
test_sac_speed[reduce-overhead-backward] 1.8118ms 1.7629ms 567.2572 Ops/s 616.7402 Ops/s $\textbf{\color{#d91a1a}-8.02\%}$
test_redq_speed[False-None] 8.0444ms 7.5573ms 132.3216 Ops/s 129.6269 Ops/s $\color{#35bf28}+2.08\%$
test_redq_speed[False-backward] 12.1559ms 11.7998ms 84.7474 Ops/s 86.0897 Ops/s $\color{#d91a1a}-1.56\%$
test_redq_speed[True-None] 2.3557ms 2.2952ms 435.6995 Ops/s 431.9277 Ops/s $\color{#35bf28}+0.87\%$
test_redq_speed[True-backward] 4.6155ms 4.1669ms 239.9842 Ops/s 247.0880 Ops/s $\color{#d91a1a}-2.88\%$
test_redq_speed[reduce-overhead-None] 2.3756ms 2.3205ms 430.9333 Ops/s 423.2513 Ops/s $\color{#35bf28}+1.81\%$
test_redq_speed[reduce-overhead-backward] 4.2564ms 4.1697ms 239.8255 Ops/s 244.2160 Ops/s $\color{#d91a1a}-1.80\%$
test_redq_deprec_speed[False-None] 9.3657ms 9.0438ms 110.5726 Ops/s 109.5633 Ops/s $\color{#35bf28}+0.92\%$
test_redq_deprec_speed[False-backward] 12.7933ms 12.3538ms 80.9467 Ops/s 82.5315 Ops/s $\color{#d91a1a}-1.92\%$
test_redq_deprec_speed[True-None] 2.6728ms 2.6012ms 384.4353 Ops/s 381.5771 Ops/s $\color{#35bf28}+0.75\%$
test_redq_deprec_speed[True-backward] 4.4342ms 4.2798ms 233.6554 Ops/s 222.9328 Ops/s $\color{#35bf28}+4.81\%$
test_redq_deprec_speed[reduce-overhead-None] 2.6848ms 2.6026ms 384.2314 Ops/s 377.9758 Ops/s $\color{#35bf28}+1.66\%$
test_redq_deprec_speed[reduce-overhead-backward] 4.3820ms 4.2748ms 233.9284 Ops/s 222.7592 Ops/s $\textbf{\color{#35bf28}+5.01\%}$
test_td3_speed[False-None] 8.0311ms 7.9739ms 125.4099 Ops/s 125.6512 Ops/s $\color{#d91a1a}-0.19\%$
test_td3_speed[False-backward] 10.9540ms 10.3136ms 96.9596 Ops/s 95.1056 Ops/s $\color{#35bf28}+1.95\%$
test_td3_speed[True-None] 1.7068ms 1.6444ms 608.1090 Ops/s 619.9861 Ops/s $\color{#d91a1a}-1.92\%$
test_td3_speed[True-backward] 3.3917ms 3.3102ms 302.1008 Ops/s 298.9696 Ops/s $\color{#35bf28}+1.05\%$
test_td3_speed[reduce-overhead-None] 49.2010ms 25.1403ms 39.7768 Ops/s 39.6663 Ops/s $\color{#35bf28}+0.28\%$
test_td3_speed[reduce-overhead-backward] 1.4493ms 1.4067ms 710.8831 Ops/s 666.0064 Ops/s $\textbf{\color{#35bf28}+6.74\%}$
test_cql_speed[False-None] 17.2398ms 16.6782ms 59.9585 Ops/s 59.6770 Ops/s $\color{#35bf28}+0.47\%$
test_cql_speed[False-backward] 22.5973ms 21.8412ms 45.7850 Ops/s 44.9900 Ops/s $\color{#35bf28}+1.77\%$
test_cql_speed[True-None] 3.3360ms 3.2129ms 311.2440 Ops/s 310.6250 Ops/s $\color{#35bf28}+0.20\%$
test_cql_speed[True-backward] 6.0572ms 5.5371ms 180.6007 Ops/s 177.7715 Ops/s $\color{#35bf28}+1.59\%$
test_cql_speed[reduce-overhead-None] 20.8685ms 12.8486ms 77.8295 Ops/s 77.4270 Ops/s $\color{#35bf28}+0.52\%$
test_cql_speed[reduce-overhead-backward] 2.1538ms 1.9658ms 508.7063 Ops/s 501.6721 Ops/s $\color{#35bf28}+1.40\%$
test_a2c_speed[False-None] 3.2409ms 3.1395ms 318.5172 Ops/s 314.4224 Ops/s $\color{#35bf28}+1.30\%$
test_a2c_speed[False-backward] 6.8962ms 6.3264ms 158.0686 Ops/s 157.5762 Ops/s $\color{#35bf28}+0.31\%$
test_a2c_speed[True-None] 1.4028ms 1.3151ms 760.3917 Ops/s 759.4623 Ops/s $\color{#35bf28}+0.12\%$
test_a2c_speed[True-backward] 3.1999ms 3.0279ms 330.2579 Ops/s 322.4994 Ops/s $\color{#35bf28}+2.41\%$
test_a2c_speed[reduce-overhead-None] 15.4062ms 8.7218ms 114.6558 Ops/s 114.7866 Ops/s $\color{#d91a1a}-0.11\%$
test_a2c_speed[reduce-overhead-backward] 1.7265ms 1.5897ms 629.0608 Ops/s 623.2858 Ops/s $\color{#35bf28}+0.93\%$
test_ppo_speed[False-None] 3.7550ms 3.6554ms 273.5702 Ops/s 267.9651 Ops/s $\color{#35bf28}+2.09\%$
test_ppo_speed[False-backward] 7.4498ms 7.0336ms 142.1743 Ops/s 141.6048 Ops/s $\color{#35bf28}+0.40\%$
test_ppo_speed[True-None] 1.4643ms 1.3841ms 722.4655 Ops/s 716.0579 Ops/s $\color{#35bf28}+0.89\%$
test_ppo_speed[True-backward] 3.3537ms 3.2209ms 310.4718 Ops/s 325.6776 Ops/s $\color{#d91a1a}-4.67\%$
test_ppo_speed[reduce-overhead-None] 1.0247ms 0.9475ms 1.0554 KOps/s 1.0607 KOps/s $\color{#d91a1a}-0.49\%$
test_ppo_speed[reduce-overhead-backward] 1.6788ms 1.5432ms 647.9980 Ops/s 697.2305 Ops/s $\textbf{\color{#d91a1a}-7.06\%}$
test_reinforce_speed[False-None] 2.4544ms 2.2435ms 445.7387 Ops/s 441.7180 Ops/s $\color{#35bf28}+0.91\%$
test_reinforce_speed[False-backward] 3.7967ms 3.3648ms 297.1971 Ops/s 305.6702 Ops/s $\color{#d91a1a}-2.77\%$
test_reinforce_speed[True-None] 1.3320ms 1.2634ms 791.4853 Ops/s 786.8663 Ops/s $\color{#35bf28}+0.59\%$
test_reinforce_speed[True-backward] 3.0901ms 3.0291ms 330.1298 Ops/s 326.9349 Ops/s $\color{#35bf28}+0.98\%$
test_reinforce_speed[reduce-overhead-None] 18.1465ms 9.7801ms 102.2483 Ops/s 103.9001 Ops/s $\color{#d91a1a}-1.59\%$
test_reinforce_speed[reduce-overhead-backward] 1.7181ms 1.6204ms 617.1473 Ops/s 666.9395 Ops/s $\textbf{\color{#d91a1a}-7.47\%}$
test_iql_speed[False-None] 9.6264ms 9.1705ms 109.0456 Ops/s 107.5899 Ops/s $\color{#35bf28}+1.35\%$
test_iql_speed[False-backward] 13.5415ms 13.1158ms 76.2436 Ops/s 76.9401 Ops/s $\color{#d91a1a}-0.91\%$
test_iql_speed[True-None] 2.3250ms 2.1731ms 460.1622 Ops/s 450.4414 Ops/s $\color{#35bf28}+2.16\%$
test_iql_speed[True-backward] 5.3186ms 4.8892ms 204.5316 Ops/s 203.5476 Ops/s $\color{#35bf28}+0.48\%$
test_iql_speed[reduce-overhead-None] 0.4860s 12.4377ms 80.4006 Ops/s 92.4986 Ops/s $\textbf{\color{#d91a1a}-13.08\%}$
test_iql_speed[reduce-overhead-backward] 2.0928ms 2.0192ms 495.2365 Ops/s 533.3500 Ops/s $\textbf{\color{#d91a1a}-7.15\%}$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 7.5364ms 6.0556ms 165.1371 Ops/s 162.3928 Ops/s $\color{#35bf28}+1.69\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.6725ms 0.3247ms 3.0796 KOps/s 3.7184 KOps/s $\textbf{\color{#d91a1a}-17.18\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.4836ms 0.3073ms 3.2538 KOps/s 4.0602 KOps/s $\textbf{\color{#d91a1a}-19.86\%}$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 6.0418ms 5.8396ms 171.2443 Ops/s 169.9883 Ops/s $\color{#35bf28}+0.74\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 1.9769ms 0.2997ms 3.3370 KOps/s 3.6091 KOps/s $\textbf{\color{#d91a1a}-7.54\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.5798ms 0.3043ms 3.2859 KOps/s 3.5068 KOps/s $\textbf{\color{#d91a1a}-6.30\%}$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000] 1.5689ms 1.3601ms 735.2549 Ops/s 736.6287 Ops/s $\color{#d91a1a}-0.19\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000] 1.6196ms 1.1857ms 843.4137 Ops/s 842.5149 Ops/s $\color{#35bf28}+0.11\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 6.0645ms 5.9644ms 167.6615 Ops/s 164.2209 Ops/s $\color{#35bf28}+2.10\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 2.0618ms 0.4076ms 2.4537 KOps/s 2.4425 KOps/s $\color{#35bf28}+0.46\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.6730ms 0.4137ms 2.4175 KOps/s 2.2706 KOps/s $\textbf{\color{#35bf28}+6.47\%}$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 6.2228ms 5.8208ms 171.7972 Ops/s 167.0965 Ops/s $\color{#35bf28}+2.81\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.7988ms 0.3216ms 3.1094 KOps/s 2.9225 KOps/s $\textbf{\color{#35bf28}+6.40\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.5909ms 0.3172ms 3.1523 KOps/s 3.0491 KOps/s $\color{#35bf28}+3.39\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 6.0643ms 5.7794ms 173.0279 Ops/s 169.4866 Ops/s $\color{#35bf28}+2.09\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 0.7352ms 0.2960ms 3.3780 KOps/s 3.2339 KOps/s $\color{#35bf28}+4.45\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.5103ms 0.2610ms 3.8312 KOps/s 3.3255 KOps/s $\textbf{\color{#35bf28}+15.21\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 6.2339ms 5.9869ms 167.0315 Ops/s 163.8153 Ops/s $\color{#35bf28}+1.96\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 1.7358ms 0.4100ms 2.4393 KOps/s 2.1626 KOps/s $\textbf{\color{#35bf28}+12.79\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.6542ms 0.4129ms 2.4219 KOps/s 2.1958 KOps/s $\textbf{\color{#35bf28}+10.30\%}$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 7.1233ms 5.5040ms 181.6849 Ops/s 176.6496 Ops/s $\color{#35bf28}+2.85\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] 10.5971ms 2.1713ms 460.5610 Ops/s 420.8839 Ops/s $\textbf{\color{#35bf28}+9.43\%}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] 7.0663ms 1.2773ms 782.8987 Ops/s 808.3582 Ops/s $\color{#d91a1a}-3.15\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] 0.4572s 14.6183ms 68.4073 Ops/s 173.8946 Ops/s $\textbf{\color{#d91a1a}-60.66\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] 5.9146ms 2.0820ms 480.3026 Ops/s 416.9683 Ops/s $\textbf{\color{#35bf28}+15.19\%}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] 10.0405ms 1.3082ms 764.3905 Ops/s 824.5531 Ops/s $\textbf{\color{#d91a1a}-7.30\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] 9.0115ms 5.7266ms 174.6238 Ops/s 30.7463 Ops/s $\textbf{\color{#35bf28}+467.95\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] 8.5168ms 2.1877ms 457.0987 Ops/s 520.2805 Ops/s $\textbf{\color{#d91a1a}-12.14\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] 8.6054ms 1.3999ms 714.3198 Ops/s 794.8954 Ops/s $\textbf{\color{#d91a1a}-10.14\%}$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-True] 13.4354ms 13.1151ms 76.2478 Ops/s 71.5752 Ops/s $\textbf{\color{#35bf28}+6.53\%}$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-False] 18.3647ms 16.6681ms 59.9950 Ops/s 60.0237 Ops/s $\color{#d91a1a}-0.05\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-True] 18.6450ms 17.9069ms 55.8444 Ops/s 53.4228 Ops/s $\color{#35bf28}+4.53\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-False] 18.6918ms 16.7111ms 59.8406 Ops/s 59.0966 Ops/s $\color{#35bf28}+1.26\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-True] 18.1409ms 17.7821ms 56.2362 Ops/s 54.1345 Ops/s $\color{#35bf28}+3.88\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-False] 19.8774ms 17.8625ms 55.9831 Ops/s 54.0798 Ops/s $\color{#35bf28}+3.52\%$

@vmoens
Update
566c09f 
[ghstack-poisoned]
@vmoens
Update
e8cd9f9 
[ghstack-poisoned]
mikaylagawarecki
mikaylagawarecki reviewed Feb 28, 2025
View reviewed changes
torchrl/envs/custom/llm.py
class LLMEnv(EnvBase):
"""A text generation environment.

This environment is designed to work with language models, where the observation is a string or a tensor of
Copy link

@mikaylagawarecki mikaylagawarecki Feb 28, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mentioning that

  • by default this is meant to track history for a prompt and users can append transforms to tailor this to their use case (e.g. CoT etc.)
  • users must append a transform to set the "done" condition, which would trigger the loading of the next prompt

might be helpful (is my understanding here correct?)

vmoens and others added 2 commits March 3, 2025 08:50
@vmoens @mikaylagawarecki
Apply suggestions from code review
f72153c 
Co-authored-by: mikaylagawarecki <[email protected]>
@vmoens
Update
c062741 
[ghstack-poisoned]
This was referenced Mar 3, 2025
Merged
Merged
Merged
@vmoens vmoens merged commit c062741 into gh/vmoens/96/base Mar 3, 2025
69 of 72 checks passed
vmoens added a commit that referenced this pull request Mar 3, 2025
@vmoens
[Feature] LLMEnv and DataLoadingPrimer
433d0e6 
ghstack-source-id: 1ad4943
Pull Request resolved: #2818
@vmoens vmoens deleted the gh/vmoens/96/head branch March 3, 2025 15:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mikaylagawarecki mikaylagawarecki mikaylagawarecki left review comments

Assignees
No one assigned
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@vmoens @mikaylagawarecki @facebook-github-bot