Change non_eos_penalty to be consistent across OnPolicy trainers

[RylanSchaeffer]

What does this PR do?

This PR is designed to address this issue: #2012

To quickly recap, PPOv2Trainer and RLOOTrainer replace non-EOS outputs' scores with a constant penalty, whereas OnlineDPOTrainer subtracts non-EOS outputs' scores by a constant penalty. After discussing with Quentin, I believe we want PPOv2Trainer and RLOOTrainer to be consistent with OnlineDPOTrainer.

Before submitting

• [x ] Was this discussed/approved via a GitHub issue? Please add a link
  to it if that's the case.
• [x ] Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines.
• Did you write any new necessary tests?

No

Who can review?

@qgallouedec

[RylanSchaeffer]

@qgallouedec I think this is ready for your review. Can you please have a look and get back to me on any additional changes you want made? Thank you!

[qgallouedec]

Thanks a lot, this PR makes sense.
One remark though, the penalty should be positive, because it's substracted.

[qgallouedec]

Before merging, I'd like to make sure that the results are still comparable. Not for all trainers, maybe just for RLOO? Do you have ressource to run an experiment?

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[RylanSchaeffer]

@qgallouedec I fixed the incorrect negative defaults. I will now run RLOO before and after the change. Does that sound like a sufficient comparison?

[RylanSchaeffer]

(Old) Command to Replace Scores:

python -u examples/scripts/ppo/ppo_tldr.py \
--learning_rate 3e-6 \
--output_dir models/minimal/ppo \
--per_device_train_batch_size 8 \
--gradient_accumulation_steps 16 \
--total_episodes 30000 \
--model_name_or_path EleutherAI/pythia-1b-deduped \
--sft_model_path cleanrl/EleutherAI_pythia-1b-deduped__sft__tldr \
--reward_model_path cleanrl/EleutherAI_pythia-1b-deduped__reward__tldr \
--non_eos_penalty \
--stop_token eos \
--response_length 53

W&B Run: https://wandb.ai/rylan/huggingface/runs/3vk55y9v

New Command to Subtract Scores:

python -u examples/scripts/ppo/ppo_tldr.py \
--learning_rate 3e-6 \
--output_dir models/minimal/ppo \
--per_device_train_batch_size 8 \
--gradient_accumulation_steps 16 \
--total_episodes 30000 \
--model_name_or_path EleutherAI/pythia-1b-deduped \
--sft_model_path cleanrl/EleutherAI_pythia-1b-deduped__sft__tldr \
--reward_model_path cleanrl/EleutherAI_pythia-1b-deduped__reward__tldr \
--missing_eos_penalty 1.0 \
--stop_token eos \
--response_length 53

W&B Run: https://wandb.ai/rylan/huggingface/runs/9l8fvykd

Results

@qgallouedec How long would you like me to let these two run for?

[RylanSchaeffer]

Subtracting and replacing seem relatively consistent with one another.

[qgallouedec]

Very nice, thanks a lot @RylanSchaeffer