support loss function for Self-play Preference Optimization

[winglian]

arxiv: https://arxiv.org/abs/2405.00675

[younesbelkada]

Thanks a lot for this great addition !
Can you add a section here in the docs to mention this method: https://github.com/huggingface/trl/blob/main/docs/source/dpo_trainer.mdx#loss-functions ! 🙏

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[winglian]

@younesbelkada docs updated. thanks!

[younesbelkada]

Thanks a lot !

[younesbelkada]

cc @kashif wdyt? 🙏

[kashif]

thanks @winglian can you kindly update the Config's doc and arguments

trl/trl/trainer/dpo_config.py

Line 72 in e2189c9

loss_type: Literal["sigmoid", "hinge", "ipo", "kto_pair", "bco_pair"] = "sigmoid"

[kashif]

@winglian do you want to add an option to the test? e.g. a ["gpt2", "sppo", True],

[angelahzyuan]

@winglian Thanks for adding our work! @younesbelkada @kashif Just submitted a new pull request at #1615. This updates the loss function according to Equation (4.8), with $P(y_w > y_l) = 1$ and $P(y_l > y_w) = 0$, and justified it in doc as the hard label version of the algorithm.

It should work well now for the first iteration. Our reported 3 iterations results was based on the soft label version.

[flozi00]

I just gave it a try and it's working better than orpo for me now.
Just installed the main branch this evening with the follow up patch for the hard loss type.

[RL4LLM]

Hi @winglian @flozi00 Do you know what is the value of beta shoud I set for SPPO?