fix 8-bit multi-gpu training bug #1353
Conversation
There was a problem hiding this comment.
Thanks very much @fancyerii ! I left one comment about making gradient_checkpointing_kwargs configurable, I think after that we can ship your PR ! 🚀
| device_map_local_process_idx: Optional[bool] = field( | ||
| default=True, metadata={"help": "whether to device map model to local process index, see " | ||
| "https://github.com/huggingface/trl/issues/1348"} | ||
| ) |
There was a problem hiding this comment.
| device_map_local_process_idx: Optional[bool] = field( | |
| default=True, metadata={"help": "whether to device map model to local process index, see " | |
| "https://github.com/huggingface/trl/issues/1348"} | |
| ) |
I think we can remove that and always create device_map = "device_map": {"": Accelerator().local_process_index}
Can you just make gradient_checkpointing_kwargs configurable here? 🙏
make gradient_checkpointing_kwargs configurable.
remote unnecessary config of device_map
|
Thanks @fancyerii ! |
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
done. |
* fix 8-bit multi-gpu training bug see huggingface#1348 * Update dpo_llama2.py make gradient_checkpointing_kwargs configurable. * Update dpo_llama2.py remote unnecessary config of device_map * format with make precommit --------- Co-authored-by: ubuntu <[email protected]>
see #1348