🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/2215

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

• Rebase your PRs: Unstable CUDA signal in CI caused by cudnn 9 update

❌ 4 New Failures

As of commit 40f7b2d with merge base 0813dc0 ():

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@wertyuilife2 can you confirm that this makes sense?

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of CPU Benchmark Tests

Total Benchmarks: 91. Improved: $\large\color{#35bf28}3$. Worsened: $\large\color{#d91a1a}2$.

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of GPU Benchmark Tests

Total Benchmarks: 94. Improved: $\large\color{#35bf28}3$. Worsened: $\large\color{#d91a1a}3$.

@vmoens I did some further search, found that many open-source libraries implementing PER maintain the historical maximum priority, such as dopamine.

But the original PER paper and some other source codes in my domain use the buffer's maximum priority approach,
including EfficientZero, whose v2 is the SOTA data-efficient method in RL.

So, overall, both approaches make sense, and this is not a bug(my bad), I believe the buffer's maximum priority approach is more robust to the priority value.

I am raising the issue because I found in practice that during the early stages of training, when a transition is first time being sampled, its PER weight is typically 1e-8 (which is the value of epsilon) which is weird. This is because max_priority=1 and the bellman error I get in the early training stage is 0 (cause network output are init to 0 for stablility).

So, basically, this is more like an additional feature for better priority adaptation or similar to priority normalization. I think it is not an essential feature (if you find implementing it to be complicated), but it can be available as an option, I believe it can increase sample efficiency and training stability.

So should I make the erasing of the max_priority during extend optional?

yep, I think so. Maybe in some task the historical max will be better.