Skip to content

[bug] objective/entropy < 0 when using rlootrainer and ppotrainer #2496

New issue
Jump to bottom
New issue
Jump to bottom
Open
Open
[bug] objective/entropy < 0 when using rlootrainer and ppotrainer#2496
Labels
❓ questionSeeking clarification or more information🏋 PPORelated to PPO🏋 RLOORelated to RLOO🙋 help from community wantedOpen invitation for community members to contribute
@macheng6

Description

@macheng6
macheng6
opened on Dec 17, 2024

trl/trl/trainer/rloo_trainer.py

Line 443 in 1661bc2

mean_entropy = (-logprobs).sum(1).mean()

This is because in the previous code, the padding part of logprod is filled with 1.

INVALID_LOGPROB=1.0
logprobs = torch.masked_fill(logprobs, padding_mask, INVALID_LOGPROB)

I don't know why INVALID_LOGPROB is set to 1, wouldn't it work fine if it is set to 0?

Metadata

Assignees

No one assigned

    Labels

    ❓ questionSeeking clarification or more information🏋 PPORelated to PPO🏋 RLOORelated to RLOO🙋 help from community wantedOpen invitation for community members to contribute

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions