Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BugFix] QValue modules and nested action #1351

Merged
merged 13 commits into from
Jul 5, 2023

Conversation

matteobettini
Copy link
Contributor

@matteobettini matteobettini commented Jul 3, 2023

Fixes #1273

Signed-off-by: Matteo Bettini <matbet@meta.com>
@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jul 3, 2023
if "action" in spec_keys:
_key = "action"
else:
# the first key is the action
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we do not find an action entry, we pick the first leaf (as done in EnvBase to get an action_key)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think it makes sense? Do we want to be more restrictive?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could only accept key that end with "action" as leaf.
up to you, i maybe slightly prefer the current version

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have modified it so that it will look for a leaf "action" key

spec_keys = action_space.keys(True, True)
if "action" in spec_keys:
_key = "action"
else:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

@matteobettini matteobettini marked this pull request as draft July 3, 2023 14:59
@matteobettini
Copy link
Contributor Author

matteobettini commented Jul 3, 2023

So @vmoens I have a proposition here.

This all business of allowing both action_space and spec to be specs is very confusing to me.

What about having action_space only be a str and spec only be a spec.

If so a user which wants to pass a str can pass it to action_space, but a user that passes a spec passes it always to spec.

Currently we allow to feed a spec to either

Signed-off-by: Matteo Bettini <matbet@meta.com>
@vmoens
Copy link
Contributor

vmoens commented Jul 3, 2023

So @vmoens I have a proposition here.

This all business of allowing both action_space and spec to be specs is very confusing to me.

What about having action_space only be a str and spec only be a spec.

If so a user which wants to pass a str can pass it to action_space, but a user that passes a spec passes it always to spec.

Currently we allow to feed a spec to either

We can make the change but unfortunately as of now I'm not open to bc breaking changes without warning, ie if we do that we must ensure that users will only see a warning until 0.3 (since this will be in 0.2).

@matteobettini
Copy link
Contributor Author

Sure, I can put the warning

Signed-off-by: Matteo Bettini <matbet@meta.com>
@matteobettini matteobettini marked this pull request as ready for review July 4, 2023 08:05
Signed-off-by: Matteo Bettini <matbet@meta.com>
@matteobettini matteobettini marked this pull request as draft July 4, 2023 08:45
Signed-off-by: Matteo Bettini <matbet@meta.com>
Signed-off-by: Matteo Bettini <matbet@meta.com>
@matteobettini matteobettini marked this pull request as ready for review July 4, 2023 10:15
Copy link
Contributor

@vmoens vmoens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM only some minor comments

test/test_actors.py Outdated Show resolved Hide resolved
torchrl/modules/tensordict_module/actors.py Outdated Show resolved Hide resolved
if "action" in spec_keys:
_key = "action"
else:
# the first key is the action
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think it makes sense? Do we want to be more restrictive?

Signed-off-by: Matteo Bettini <matbet@meta.com>
Signed-off-by: Matteo Bettini <matbet@meta.com>
Signed-off-by: Matteo Bettini <matbet@meta.com>
Signed-off-by: Matteo Bettini <matbet@meta.com>
Signed-off-by: Matteo Bettini <matbet@meta.com>
Copy link
Contributor

@vmoens vmoens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM good to merge

@vmoens vmoens merged commit e09d2b3 into pytorch:main Jul 5, 2023
@matteobettini matteobettini deleted the parametric_modules branch July 5, 2023 13:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG] Is _process_action_space_spec compatible with nested keys?
3 participants