Code

Add LLaMA 3 Python support#725

Code

Merged

karpathy merged 36 commits intokarpathy:masterkarpathy/llm.c:masterfrom gordicaleksa:llamagordicaleksa/llm.c:llamaCopy head branch name to clipboardAug 8, 2024

+1,284

Commits

Commits on Aug 2, 2024

• Equivalent with nano llama 3

  gordicaleksa

  committed
  41bf8e0

  Copy full SHA for 41bf8e0

• Refactor

  gordicaleksa

  committed
  838cd13

  Copy full SHA for 838cd13

• Minor refactor

  gordicaleksa

  committed
  c414d02

  Copy full SHA for c414d02

Commits on Aug 3, 2024

• Equivalent to nano llama 3 reference code

  gordicaleksa

  committed
  465aac4

  Copy full SHA for 465aac4

• Refactor attn, change numerics but equivalent

  gordicaleksa

  committed
  f50f2de

  Copy full SHA for f50f2de

• Have prompts in a file instead of inline, prompt 4 is different

  gordicaleksa

  committed
  c0c08ba

  Copy full SHA for c0c08ba

• Refactor checkpoint state dict map func

  gordicaleksa

  committed
  de879d1

  Copy full SHA for de879d1

• Refactor MLP

  gordicaleksa

  committed
  0199e51

  Copy full SHA for 0199e51

• Refactor attn mechanism

  gordicaleksa

  committed
  fdd5345

  Copy full SHA for fdd5345

• One more minor attn fix

  gordicaleksa

  committed
  fa7bcc3

  Copy full SHA for fa7bcc3

• Unify generate and generate_llama

  gordicaleksa

  committed
  180215f

  Copy full SHA for 180215f

• Fix generate for gpt-2

  gordicaleksa

  committed
  8919b66

  Copy full SHA for 8919b66

• Going towards pure llama 3 file - fixed attn

  gordicaleksa

  committed
  ccdbdfd

  Copy full SHA for ccdbdfd

• MLP GPT2->LLaMA3

  gordicaleksa

  committed
  8a48df7

  Copy full SHA for 8a48df7

• Removed from pretrained for GPT-2

  gordicaleksa

  committed
  c1d2b7f

  Copy full SHA for c1d2b7f

• Refactoring - got to main

  gordicaleksa

  committed
  d855c96

  Copy full SHA for d855c96

• Got to llama 3 inference (end)

  gordicaleksa

  committed
  b1acb59

  Copy full SHA for b1acb59

• Done - need to test train loop and saving model

  gordicaleksa

  committed
  bad7857

  Copy full SHA for bad7857

Commits on Aug 4, 2024

• Remove init weights as it's gpt-2 specific

  gordicaleksa

  committed
  879cc5f

  Copy full SHA for 879cc5f

• Add prompts file

  gordicaleksa

  committed
  7768a36

  Copy full SHA for 7768a36

• Fix saving model / state logic

  gordicaleksa

  committed
  cd90273

  Copy full SHA for cd90273

• Test training loop works

  gordicaleksa

  committed
  4b386a2

  Copy full SHA for 4b386a2

• Minor refactor - remove wpe pos array from fwd

  gordicaleksa

  committed
  0749a4a

  Copy full SHA for 0749a4a

• Support HF & Meta models

  gordicaleksa

  committed
  8e55d16

  Copy full SHA for 8e55d16

• Remove float(-inf)

  gordicaleksa

  committed
  72dcfeb

  Copy full SHA for 72dcfeb

Commits on Aug 8, 2024

• Remove llmc_py, single file

  gordicaleksa

  committed
  d4ef9c5

  Copy full SHA for d4ef9c5

• Add explicit external mask

  gordicaleksa

  committed
  b25e325

  Copy full SHA for b25e325

• Add llama config error check

  gordicaleksa

  committed
  b7c98c9

  Copy full SHA for b7c98c9

• Rename the new file to train llama3

  gordicaleksa

  committed
  624ed3c

  Copy full SHA for 624ed3c

• Remove prompts.json

  gordicaleksa

  committed
  dfd459b

  Copy full SHA for dfd459b

• Remove the whole llmc_py

  gordicaleksa

  committed
  ac01536

  Copy full SHA for ac01536

• Remove pycache

  gordicaleksa

  committed
  89addd3

  Copy full SHA for 89addd3

• Address Andrej's PR comments

  gordicaleksa

  committed
  f1c91f8

  Copy full SHA for f1c91f8

• Add data loader not implemented exception

  gordicaleksa

  committed
  8b672ff

  Copy full SHA for 8b672ff

• Add comments, fix stop tokens

  gordicaleksa

  committed
  c5c87fc

  Copy full SHA for c5c87fc

• Remove unnecessary comment

  gordicaleksa

  committed
  d773c88

  Copy full SHA for d773c88