[KTO] fix tokenization bugs

[kawine]

• fixed the tokenization bugs mentioned here Bos token in prompt_input_ids but not in completion_input_ids? #1401
• made some minor changes to the doc to clarify this issue KTOTrainer vs kto_loss in DPO-Trainer #1387

cc @kashif

[kashif]

thanks @kawine checking now

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[lewtun]

Thanks a lot for the fix @kawine ! Would you mind adding a small unit test for the tokenize_row() method to ensure we don't regress on this in future 🙏 ?

[kashif]

@kawine you can use this test:

    def test_kto_trainer_tokenize_row(self):
        with tempfile.TemporaryDirectory() as tmp_dir:
            training_args = KTOConfig(
                output_dir=tmp_dir,
                per_device_train_batch_size=2,
                max_steps=3,
                remove_unused_columns=False,
                gradient_accumulation_steps=1,
                learning_rate=9e-1,
                evaluation_strategy="steps",
                beta=0.1,
            )

            dummy_dataset = self._init_dummy_dataset()

            trainer = KTOTrainer(
                model=self.model,
                ref_model=self.ref_model,
                args=training_args,
                tokenizer=self.tokenizer,
                train_dataset=dummy_dataset,
                eval_dataset=dummy_dataset,
            )

            row = dummy_dataset[0]

            # test that the row can be tokenized
            tokenized_row = trainer.tokenize_row(row)

            # Assert bos_token_id
            assert tokenized_row["prompt_input_ids"][0] == self.tokenizer.bos_token_id
            assert tokenized_row["completion_input_ids"][0] == self.tokenizer.bos_token_id

[kawine]

thanks! test has been added @kashif