Handle last token from generation prompt

[pablovicente]

Some tokenizers, like Llama BPE tokenizer, might merge tokens into one such as the case where there is a space follow by some characters like $.

In a previous PR we handled the case for chosen/rejected but one small improvement is to remove the extra space token from the end of the prompt used for generation in get_batch_samples. It should left to the model if the next generated token is just the space token or one such that it combines the space with other token into one. To do so, we need to manually check prompt_ids for chosen/rejected and by itself and chose the shortest knowing that they can only differ by 1 token.

Given the following example, prompt + chosen will leave the space after [\INST] unchanged but it wont be the case for prompt+rejected since it starts with $.

"prompt": "[INST] How is the stock price? [/INST] "
"chosen": "46 as of 10am EST"
"rejected":  "$46 as of 10am EST"

@kashif

[kashif]

@pablovicente are the tests passing for you?

[pablovicente]

@pablovicente are the tests passing for you?

test_dpo_trainer.py tests pass. I see issues on the sft_trainer but have not made changes there. Are dpo_trainer tests pass for you?

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[younesbelkada]

@kashif we can merge this no? Wdyt? 🙏