[XPO] xpo trainer

[kashif]

https://huggingface.co/papers/2405.21046 implementation

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[qgallouedec]

kl isn't logged in XPO. Is it possible to have it logged?

[qgallouedec]

Is it possible to have it logged?

from @kashif

In the online-dpo what is the "kl" being logged?

"kl" is the approximate KL div from the ref model distribution to the trained model distribution. Approximated as:

trl/trl/trainer/online_dpo_trainer.py

Line 389 in 72f19c3

kl = logprobs - ref_logprobs

Your calculation abdf1a5 seems correct to me

        # Calculate KL divergence for model and ref data
         kl_model_data = model_logprobs_model_data - ref_logprobs_model_data
         kl_ref_data = model_logprobs_ref_data - ref_logprobs_ref_data
         mean_kl = (kl_model_data.sum(1) + kl_ref_data.sum(1)).mean() / 2
         self.stats["objective/kl"].append(gather_mean(mean_kl))

[qgallouedec]

Very nice work @kashif