VLM dpo bug

[liuchaohu]

trl/trainer/dpo_trainer.py line 542
The tokenizer for super().init () should be set to self.tokenizer instead of tokenizer, otherwise the previous is_vision_model will be invalid.

[qgallouedec]

what do you mean by

otherwise the previous is_vision_model will be invalid.

?

When the model is a VLM, the transformers.Trainer expect a processor for the arg tokenizer, not a tokenizer.
in other words:

Trainer(
    ...,
    # tokenizer=processor.tokenizer, # NO
    tokenizer=processor, # YES
)

[liuchaohu]

In line 341-343, if self.is_vision_model == True, then self.processor = tokenizer(i.e., processor) and self.tokenizer = tokenizer.tokenizer.
However, In line 536-548, super().init(...) will set self.tokenizer = tokenizer(i.e., processor).
In this case, both self.tokenizer & self.processor are tokenizer(i.e., processor).

[qgallouedec]

We are refactoring DPO data processing. The new implementation should be more readable.