When you backprop the two losses, I guess that in your implementation the gradients for the common modules are a sum of gradient for the two losses, right?
What compare_against does is that it feeds the module (in this case the critic) with a detached version of the params used for the other module (in this case the policy). This is what is done in some algos but I understand that it may be suboptimal in other cases, especially if the reset of your critic is very shallow.

Have you tried to just remove the compare_against?

If that works, a solution could be an optional call compare_against with a default to not doing it.

Some updates. I actually only need to modify this line: https://github.com/pytorch/rl/blob/main/torchrl/objectives/ppo.py#L117

Here is what works and what does not:
self.convert_to_functional(critic, "critic", compare_against=self.actor_params) # Value network update is not correct
self.convert_to_functional(critic, "critic") # Value network update is not correct either
self.critic = critic # This works

So removing the compare_against it does not seem to be the solution.

When you backprop the two losses, I guess that in your implementation the gradients for the common modules are a sum of gradient for the two losses, right? What compare_against does is that it feeds the module (in this case the critic) with a detached version of the params used for the other module (in this case the policy). This is what is done in some algos but I understand that it may be suboptimal in other cases, especially if the reset of your critic is very shallow.

Have you tried to just remove the compare_against?

If that works, a solution could be an optional call compare_against with a default to not doing it.

Yes for the code to work the gradients have to be the sum of the two losses

You're right, even without that, there is not param tighing. Let me fix that to have a more predictable behaviour.

Closed by #1037