Seed is not applied for DPO recipes

[bogdansalyp]

TL;DR

Launching same config twice with seed: 42 results in two different loss curves

Affected recipes

full_dpo_distributed - seed is not set

Full DPO is taken from #2275

lora_dpo_distributed - seed is not set

Not affected recipes

full_finetune_distributed - works fine

lora_finetune_distributed - works fine

[acisseJZhong]

Hi @bogdansalyp, could you share more about the run information, e.g. the config and run command?

[bogdansalyp]

Hi @bogdansalyp, could you share more about the run information, e.g. the config and run command?

Yes, it's just standard llama3_1/8B_lora_dpo.yaml config but with seed: 42 instead of seed: null

I didn't test it on other recipes yet

UPD: Updated description with the recipes I've tried

[ebsmothers]

@bogdansalyp agree this looks a bit weird. One observation is that the y-axis in some of the plots is really small though, I do wonder whether you also see variation of ~1e-3 in the unaffected recipes' losses? (Because I think exact numerical parity is not achievable with bf16). For debugging, would suggest inspecting two things: (1) are the model weights the same across runs? (2) are the samples seen the same across runs? If both (1) and (2) are true, my guess would be that it's just accumulated numerical error. But for (1) in the LoRA DPO recipe especially -- we don't load in LoRA weights, so those will be randomly initialized. Definitely worth checking if they're identical across runs. (It's also possible that there's another source of randomness I haven't yet accounted for.)