Code

Full DPO Distributed#2275

Code

Merged

ebsmothers merged 39 commits intopytorch:mainpytorch/torchtune:mainfrom sam-pi:add-feature-full-dposam-pi/torchtune:add-feature-full-dpoCopy head branch name to clipboardFeb 7, 2025

+1,425-21

Commits

Commits on Jan 17, 2025

• full dpo configs, distributed recipe, and integration tests

  sam-pi

  committed
  b96255b

  Copy full SHA for b96255b

Commits on Jan 23, 2025

• disable dropout, ref model setup, minor doc update

  sam-pi

  committed
  761b718

  Copy full SHA for 761b718

Commits on Jan 28, 2025

• Merge remote-tracking branch 'upstream/main' into HEAD

  SalmanMohammadi

  committed
  753e822

  Copy full SHA for 753e822

• updating full recipe

  SalmanMohammadi

  committed
  0f90093

  Copy full SHA for 0f90093

Commits on Jan 30, 2025

• updating recipe

  SalmanMohammadi

  committed
  ebed89c

  Copy full SHA for ebed89c

• removing 70B full dpo config until multi-node support is available

  sam-pi

  committed
  aff595f

  Copy full SHA for aff595f

• minor update to avoid _ref_model self reference

  sam-pi

  committed
  431f269

  Copy full SHA for 431f269

• clean up rank zero logs and ref_checkpointer

  sam-pi

  committed
  c63e9e8

  Copy full SHA for c63e9e8

• remove unncessary save/load test and update to 2 GPUs

  sam-pi

  committed
  ebf288a

  Copy full SHA for ebf288a

• fix: Metrics weren't running and synced across devices

  bogdansalyp

  committed
  ba12bb4

  Copy full SHA for ba12bb4

Commits on Jan 31, 2025

• fix: Fixed tokens_per_second_per_gpu

  bogdansalyp

  committed
  6139096

  Copy full SHA for 6139096

• fix: Fixed torch.distributed naming

  bogdansalyp

  committed
  2a4ca92

  Copy full SHA for 2a4ca92

• fix: tokens_per_second_pre_gpu fixed for full dpo

  bogdansalyp

  committed
  7f94b07

  Copy full SHA for 7f94b07

• fix: Added running metrics to full_dpo_distributed

  bogdansalyp

  committed
  1a673df

  Copy full SHA for 1a673df

Commits on Feb 1, 2025

• Merge pull request #2 from bogdansalyp/fix/running_metrics_and_sync_lora_dpo

  Show description for 16821c4
  sam-pi

  authored
  16821c4

  Copy full SHA for 16821c4

Commits on Feb 3, 2025

• fix: num_tokens all_reduce crash in DPO recipes

  bogdansalyp

  committed
  d052271

  Copy full SHA for d052271

• Merge pull request #3 from bogdansalyp/fix/num_tokens-tensor-issue

  Show description for f9fedc4
  sam-pi

  authored
  f9fedc4

  Copy full SHA for f9fedc4

• delete ref logits and improved default full dpo config

  sam-pi

  committed
  a0ac5aa

  Copy full SHA for a0ac5aa

Commits on Feb 4, 2025

• remove 70B full DPO for now

  sam-pi

  committed
  5af4bed

  Copy full SHA for 5af4bed

• Update recipes/full_dpo_distributed.py

  Show description for 7dd27bb
  

  sam-pi

  and

  SalmanMohammadi

  authored
  7dd27bb

  Copy full SHA for 7dd27bb

• Update recipes/full_dpo_distributed.py

  Show description for 86e7639
  

  sam-pi

  and

  SalmanMohammadi

  authored
  86e7639

  Copy full SHA for 86e7639

• minor docs update and config comment

  sam-pi

  committed
  fb228c6

  Copy full SHA for fb228c6

Commits on Feb 5, 2025

• fix linting

  sam-pi

  committed
  85ce53d

  Copy full SHA for 85ce53d

Commits on Feb 6, 2025

• explicitly specify 2 GPUs for full DPO test

  sam-pi

  committed
  ad7de59

  Copy full SHA for ad7de59

• reduce dpo test VRAM usage

  sam-pi

  committed
  4e67853

  Copy full SHA for 4e67853

• Update recipes/configs/llama3_1/8B_full_dpo.yaml

  SalmanMohammadi

  authored
  4667adf

  Copy full SHA for 4667adf

• Update docs/source/recipes/dpo.rst

  SalmanMohammadi

  authored
  f2e9f47

  Copy full SHA for f2e9f47

• use get_lr, remove ac_mode, add clip_grad_norm

  sam-pi

  committed
  99a87e8

  Copy full SHA for 99a87e8

• average running loss across ranks

  sam-pi

  committed
  8e46c93

  Copy full SHA for 8e46c93

• adding opt in bwd

  SalmanMohammadi

  committed
  716efca

  Copy full SHA for 716efca

• fixing test

  SalmanMohammadi

  committed
  eef1b01

  Copy full SHA for eef1b01

• removing grad clip

  SalmanMohammadi

  committed
  46c59ec

  Copy full SHA for 46c59ec

• Updating test

  SalmanMohammadi

  committed
  6781894

  Copy full SHA for 6781894

Commits on Feb 7, 2025

• fixing test... round 2

  SalmanMohammadi

  committed
  4fd18bf

  Copy full SHA for 4fd18bf

• fixing typo

  SalmanMohammadi

  committed
  9c71083

  Copy full SHA for 9c71083

• updating distributed test to correctly resume from checkpoint

  SalmanMohammadi

  committed
  b293ce3

  Copy full SHA for b293ce3

• updating tests and recipe

  SalmanMohammadi

  committed
  893398e

  Copy full SHA for 893398e

• revert optimizer_in_bwd update for now

  sam-pi

  committed
  abbdf11

  Copy full SHA for abbdf11

• activations handling only for policy forward pass

  sam-pi

  committed
  d463e70

  Copy full SHA for d463e70