[core] officially support SFT (Supervised Finetuning)

[younesbelkada]

What does this PR do?

This PR introduces SFTTrainer class. A handy and easy to use class to train your supervised fine-tuned model on instruction-based datasets.
The API is easy to use, and also modular enough if you want to customize your training for advanced users.

You just need to pass a model id, optionally pass a PeftConfig to train adapters only. Pass also from_pretrained kwargs directly to SFTTrainer for advanced users, for example to load your model in 8bit mode.

The PR also introduces ConstantLengthDataset, a handy class to create instruction-based datasets. Just pass a tokenizer, a dataset, and a function to specify the formatting you want to have, and you should be good to go

Quickstart:

from transformers import AutoModelForCausalLM
from datasets import load_dataset
from trl import SFTTrainer

dataset = load_dataset("imdb", split="train")

model = AutoModelForCausalLM.from_pretrained("facebook/opt-350m")

trainer = SFTTrainer(
    model,
    train_dataset=dataset,
    dataset_text_field="text",
)

trainer.train()

cc @lvwerra

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[lvwerra]

Getting into great shape! Left a few comments.

[lvwerra]

i would avoid kwargs fields as much as possible. what values can be passed here? can't have them as separate kwargs?

[younesbelkada]

The dataset_kwargs corresponds to the kwargs of ConstantLengthDataset, there are 6 optional of them, I think we should move them to proper kwarg as we can always modify that class, but for prepare_int8_training kwargs I would maybe keep them as they are

[younesbelkada]

Or maybe we should educate users to create models outside the trainer in case they want to have full control over that function and remove prepare_int8_training_kwargs. Wdyt?

[lvwerra]

I think you can just tokenize the dataset so you have the input ids. i would not pad at all (the collator will do this, and if a batch has elements that are all shorter it will be faster).

then you can just pass the tokenized dataset along the data collator (data_collator = DataCollatorForLanguageModeling(tokenizer, mlm=False))

we did something similar here: https://huggingface.co/learn/nlp-course/chapter7/6?fw=pt

then we can maybe just call the function _tokenize_dataset

[younesbelkada]

Thanks for the great pointer!

[lewtun]

Thanks for adding this sweet feature @younesbelkada 🔥 !

I've left a few questions and suggestions for things that could help improve user understanding. I'll let the core maintainer approve this :)

[lewtun]

Does trl provide a helper method for this? If yes, it would be nice to see a small example in the docs of how this dataset works

[younesbelkada]

I added a pointer to an example that uses this in 91b2643

[lvwerra]

I am slightly opposed to pass kwargs like that. I think if people want to use something other than default they should just load the model outside. It's just one additional line of code for them. We on the other hand need to then worry about those: e.g. if someone has a typo in one of the above kwargs they will get a weird error because it's passed to the model.

What do you think?

[younesbelkada]

Yeah totally aligned on this!

[tigerinus]

The IMDB dataset is a sentence labelled by 1 or 0 indicating whether the sentence is positive feedback or negative feedback.

Question is, does this SFTTrainer train against the labels or the context of each sentence?

[lvwerra]

No, the SFTTrainer only trains on the text with the causal language modeling objective.

[hezhiyang2000]

Is this function suitable to train a instruction-following LLM? I reviewed the code and can't find the code about labels that avoiding to calculate the loss of instruction, prompts and labels.

[lvwerra]

See the docs here: https://huggingface.co/docs/trl/sft_trainer#train-on-completions-only