Refactor ScriptArguments

[qgallouedec]

What does this PR do?

Script arguments class is defined

• Directly in scripts, like
  

  trl/examples/scripts/orpo.py

  Lines 64 to 69 in 70036bf

  	@dataclass
  	class ScriptArguments:
  	dataset_name: str = field(
  	default="trl-internal-testing/hh-rlhf-helpful-base-trl-style",
  	metadata={"help": "The name of the dataset to use."},
  	)

• with SFTScriptArguments:
  

  trl/trl/commands/cli_utils.py

  Lines 82 to 94 in 70036bf

  	@dataclass
  	class SFTScriptArguments:
  	dataset_name: str = field(
  	default="timdettmers/openassistant-guanaco",
  	metadata={"help": "the dataset name"},
  	)
  	dataset_train_split: str = field(default="train", metadata={"help": "The dataset split to train on"})
  	dataset_test_split: str = field(default="test", metadata={"help": "The dataset split to evaluate on"})
  	config: str = field(default=None, metadata={"help": "Path to the optional config file"})
  	gradient_checkpointing_use_reentrant: bool = field(
  	default=False,
  	metadata={"help": "Whether to apply `use_reentrant` for gradient_checkpointing"},
  	)

• with DPOScriptArguments:
  

  trl/trl/commands/cli_utils.py

  Lines 112 to 129 in 70036bf

  	@dataclass
  	class DPOScriptArguments:
  	dataset_name: str = field(default=None, metadata={"help": "the dataset name"})
  	dataset_train_split: str = field(default="train", metadata={"help": "The dataset split to use for training"})
  	dataset_test_split: str = field(default="test", metadata={"help": "The dataset split to use for evaluation"})
  	ignore_bias_buffers: bool = field(
  	default=False,
  	metadata={
  	"help": "debug argument for distributed training;"
  	"fix for DDP issues with LM bias/mask buffers - invalid scalar type,`inplace operation. See"
  	"https://github.com/huggingface/transformers/issues/22482#issuecomment-1595790992"
  	},
  	)
  	config: str = field(default=None, metadata={"help": "Path to the optional config file"})
  	gradient_checkpointing_use_reentrant: bool = field(
  	default=False,
  	metadata={"help": "Whether to apply `use_reentrant` for gradient_checkpointing"},
  	)

• with RewardScriptArguments:
  

  trl/trl/commands/cli_utils.py

  Lines 97 to 109 in 70036bf

  	@dataclass
  	class RewardScriptArguments:
  	dataset_name: str = field(
  	default="trl-lib/ultrafeedback_binarized",
  	metadata={"help": "the dataset name"},
  	)
  	dataset_train_split: str = field(default="train", metadata={"help": "The dataset split to train on"})
  	dataset_test_split: str = field(default="test", metadata={"help": "The dataset split to evaluate on"})
  	config: str = field(default=None, metadata={"help": "Path to the optional config file"})
  	gradient_checkpointing_use_reentrant: bool = field(
  	default=False,
  	metadata={"help": "Whether to apply `use_reentrant` for gradient_checkpointing"},
  	)

They are pretty much the same. I suggest:

1. Merging everything into one ScriptArguments
2. Deprecate SFTScriptArguments, DPOScriptArguments and RewardScriptArguments in favor of ScriptArguments.
3. Standardize the use of ScriptArguments to all examples scripts.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a GitHub issue? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[qgallouedec]

It seems to be a debug argument. Should we remove it like in #2055?

[edbeeching]

Thanks for harmonizing all this, LGTM apart from 1 change about the split for some of the RL examples, if you change it be sure to update the docs etc.