[Feature] Add reduction parameter to Off-Policy losses. #1956

albertbou92 · 2024-02-23T11:25:33Z

Description

Same as #1890, but for Off-Policy losses.

Motivation and Context

Why is this change required? What problem does it solve?
If it fixes an open issue, please link to the issue here.
You can use the syntax close #15213 if this solves the issue #15213

I have raised an issue to propose this change (required for new features and bug fixes)

Types of changes

What types of changes does your code introduce? Remove all that do not apply:

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds core functionality)
Breaking change (fix or feature that would cause existing functionality to change)
Documentation (update in the documentation)
Example (update in the folder of examples)

Checklist

Go over all the following points, and put an x in all the boxes that apply.
If you are unsure about any of these, don't hesitate to ask. We are here to help!

I have read the CONTRIBUTION guide (required)
My change requires a change to the documentation.
I have updated the tests accordingly (required for a bug fix or a new feature).
I have updated the documentation accordingly.

pytorch-bot · 2024-02-23T11:25:37Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/1956

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 6 New Failures, 21 Unrelated Failures

As of commit 65c6399 with merge base 492091a ():

NEW FAILURES - The following jobs have failed:

Continuous Benchmark (PR) / CPU Pytest benchmark (gh)
Workflow failed! Resource not accessible by integration
Continuous Benchmark (PR) / GPU Pytest benchmark (gh)
Workflow failed! Resource not accessible by integration
Unit-tests on MacOS CPU / tests (3.11) / macos-job (gh)
The process '/usr/local/bin/git' failed with exit code 128
Unit-tests on MacOS CPU / tests (3.8) / macos-job (gh)
The process '/usr/local/bin/git' failed with exit code 128
Unit-tests on Windows / unittests-cpu / windows-job (gh)
The process 'C:\Program Files\Git\cmd\git.exe' failed with exit code 128
Unit-tests on Windows / unittests-gpu / windows-job (gh)
The process 'C:\Program Files\Git\cmd\git.exe' failed with exit code 128

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

Examples Tests on Linux / tests (3.9, 12.1) / linux-job (gh)
The process '/usr/bin/git' failed with exit code 128
Generate documentation / build-docs (3.9, 12.1) / linux-job (gh)
The process '/usr/bin/git' failed with exit code 128
Habitat Tests on Linux / tests (3.9, 11.6) / linux-job (gh)
The process '/usr/bin/git' failed with exit code 128
Libs Tests on Linux / unittests-brax (3.9, 12.1) / linux-job (gh)
The process '/usr/bin/git' failed with exit code 128
Libs Tests on Linux / unittests-gym (3.9, 12.1) / linux-job (gh)
The process '/usr/bin/git' failed with exit code 128
Libs Tests on Linux / unittests-jumanji (3.9, 12.1) / linux-job (gh)
The process '/usr/bin/git' failed with exit code 128
Libs Tests on Linux / unittests-pettingzoo / linux-job (gh)
The process '/usr/bin/git' failed with exit code 128
Libs Tests on Linux / unittests-robohive (3.9, 12.1) / linux-job (gh)
The process '/usr/bin/git' failed with exit code 128
Libs Tests on Linux / unittests-sklearn (3.9, 12.1) / linux-job (gh)
The process '/usr/bin/git' failed with exit code 128
Libs Tests on Linux / unittests-vmas (3.9, 12.1) / linux-job (gh)
The process '/usr/bin/git' failed with exit code 128
Lint / c-source / linux-job (gh)
The process '/usr/bin/git' failed with exit code 128
Lint / python-source-and-configs / linux-job (gh)
The process '/usr/bin/git' failed with exit code 128
RLHF Tests on Linux / unittests (3.9, 12.1) / linux-job (gh)
The process '/usr/bin/git' failed with exit code 128
Unit-tests on Linux / tests-cpu (3.10) / linux-job (gh)
The process '/usr/bin/git' failed with exit code 128
Unit-tests on Linux / tests-cpu (3.11) / linux-job (gh)
The process '/usr/bin/git' failed with exit code 128
Unit-tests on Linux / tests-cpu (3.8) / linux-job (gh)
The process '/usr/bin/git' failed with exit code 128
Unit-tests on Linux / tests-cpu (3.9) / linux-job (gh)
The process '/usr/bin/git' failed with exit code 128
Unit-tests on Linux / tests-gpu (3.8, 12.1) / linux-job (gh)
The process '/usr/bin/git' failed with exit code 128
Unit-tests on Linux / tests-olddeps (3.8, 11.6) / linux-job (gh)
The process '/usr/bin/git' failed with exit code 128
Unit-tests on Linux / tests-optdeps (3.9, 12.1) / linux-job (gh)
The process '/usr/bin/git' failed with exit code 128
Unit-tests on Linux / tests-stable-gpu (3.8, 11.8) / linux-job (gh)
The process '/usr/bin/git' failed with exit code 128

This comment was automatically generated by Dr. CI and updates every 15 minutes.

vmoens

Thanks so much this is an amazing feature!

We should also do the deprecated REDQ because i'll de-deprecate it soon (it runs faster than the other lol)

test/test_cost.py

albertbou92 · 2024-02-26T14:56:40Z

I have changed the test so each loss has a dedicated test for the reduction parameter. Also added the reduction parameter to deprecated REDQ.

For on-policy losses we did not have dedicated tests for reduction, we added them to the general test (e.g. test_ppo()). Should we change that too? And if so, in this PR? maybe cleaner to do it in a separate PR.

vmoens · 2024-02-26T15:23:11Z

Another PR seems appropriate! Thanks!

vmoens

Looks awesome thanks!
Approving it but can you quickly simplify the DQN case? I think we can avoid the call to apply in this case (which will make the loss faster to execute)

torchrl/objectives/dqn.py

albertbou92 added 6 commits February 23, 2024 10:32

reduction param

3ce67b9

reduction param

36c8fb0

tests

c4baf52

tests

e126e90

format

9d109d7

format

8c87f3d

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Feb 23, 2024

vmoens added the enhancement New feature or request label Feb 23, 2024

vmoens reviewed Feb 25, 2024

View reviewed changes

albertbou92 added 17 commits February 26, 2024 11:21

reduction in deprecated REDQ

1bb4371

dedicated DQN test

7af5009

dedicated DQN test

df57d77

dedicated distributional DQN test

fe0fdbc

dedicated distributional DQN test

a2102e1

dedicated TD3 test

afd1ffa

dedicated TD3 test

5d600c3

dedicated SAC test

36df6d5

dedicated discrete SAC test

5f1832c

dedicated discrete SAC test

e155e1d

dedicated discrete SAC test

7390324

dedicated ddpg test

079ff27

dedicated redq test

19150de

dedicated redq test

a989ef8

dedicated redq test

1a50ca2

sac fix

2c2e4bb

redq test fix

5b9011c

albertbou92 requested a review from vmoens February 26, 2024 14:56

vmoens approved these changes Feb 26, 2024

View reviewed changes

torchrl/objectives/dqn.py Outdated Show resolved Hide resolved

simplify dqn

3a2a768

albertbou92 requested a review from vmoens February 26, 2024 16:34

albertbou92 added 2 commits February 26, 2024 17:35

simplify dqn

ced328f

format

65c6399

vmoens approved these changes Feb 26, 2024

View reviewed changes

vmoens merged commit 6274b27 into pytorch:main Feb 26, 2024
25 of 51 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] Add reduction parameter to Off-Policy losses. #1956

[Feature] Add reduction parameter to Off-Policy losses. #1956

albertbou92 commented Feb 23, 2024

pytorch-bot bot commented Feb 23, 2024 •

edited

Loading

vmoens left a comment

albertbou92 commented Feb 26, 2024

vmoens commented Feb 26, 2024

vmoens left a comment

[Feature] Add reduction parameter to Off-Policy losses. #1956

[Feature] Add reduction parameter to Off-Policy losses. #1956

Conversation

albertbou92 commented Feb 23, 2024

Description

Motivation and Context

Types of changes

Checklist

pytorch-bot bot commented Feb 23, 2024 • edited Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/1956

❌ 6 New Failures, 21 Unrelated Failures

vmoens left a comment

Choose a reason for hiding this comment

albertbou92 commented Feb 26, 2024

vmoens commented Feb 26, 2024

vmoens left a comment

Choose a reason for hiding this comment

pytorch-bot bot commented Feb 23, 2024 •

edited

Loading