stack-llama #273
stack-llama #273
Conversation
|
The documentation is not available anymore as the PR was closed or merged. |
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
There was a problem hiding this comment.
Looks in really good! Left a few comments.
What do you think about simplifying the naming of the scripts a bit:
reward_modeling.pyrl_training.pysupervised_finetuning.py
Even if we used DeepSpeed, since with PEFT it should work as well without I think we can omit it here for simplification. What do you think?
| 1. Supervised fine-tuning of the base llama-7b model to create llama-7b-se: | ||
| - `python examples/stack_llama/scripts/sft_stack_exchange_peft.py --model_path=<LLAMA_MODEL_PATH> --streaming --no_gradient_checkpointing --learning_rate 1e-5 --max_steps 5000 --output_dir ./llama-se` | ||
| 2. Reward modeling using dialog pairs from the SE dataset using the llama-7b-se to create llama-7b-se-rm: | ||
| - `deepspeed --num_gpus=8 examples/stack_llama/scripts/reward_modeling_peft.py --model_name=<LLAMA_SE_MODEL> --deepspeed="/fsx/kashif/llama-SE/ds_config.json"` |
There was a problem hiding this comment.
We don't really need DS for this right if we use PEFT? can we just use accelerate here? I think this would make things much simpler.
There was a problem hiding this comment.
Yes I agree, just matching what was used when @kashif trained
| - `deepspeed --num_gpus=8 examples/stack_llama/scripts/reward_modeling_peft.py --model_name=<LLAMA_SE_MODEL> --deepspeed="/fsx/kashif/llama-SE/ds_config.json"` | ||
| 3. RL fine-tuning of llama-7b-se with the llama-7b-se-rm reward model: | ||
| - `acclerate launch examples/stack_llama/scripts/rl_finetuning_peft.py --log_with=wandb --model_name=<LLAMA_SE_MODEL> --reward_model_name=<LLAMA_SE_RM_MODEL> --adafactor=False --tokenizer_name=<LLAMA_TOKENIZER> --save_freq=100 --output_max_length=128 --batch_size=8 --gradient_accumulation_steps=8 --batched_gen=True --ppo_epochs=4 --seed=0 --learning_rate=1.4e-5 --early_stopping=True --output_dir=llama-se-rl-finetune-128-8-8-1.4e-5_adam` | ||
|
|
There was a problem hiding this comment.
I think we should have a note that if you want to use multiple GPUs you should use torchrun plust the appropriate args (same with accelerate).
actually we could also add the mutli-gpu commands as default since they also work for 1 GPU.
Co-authored-by: Leandro von Werra <[email protected]>
Co-authored-by: Leandro von Werra <[email protected]>
adds the stack-llama example