Code

SOTA 3-bit quants #5196

Code

Merged

ikawrakow merged 14 commits intomasterggerganov/llama.cpp:masterfrom ik/iq3_xxsggerganov/llama.cpp:ik/iq3_xxsCopy head branch name to clipboardJan 30, 2024

+1,215-18

Commits

Commits on Jan 27, 2024

• iq3_xxs: quantize/dequantize

  Show description for 8524d27
  Kawrakow

  committed
  8524d27

  Copy full SHA for 8524d27

• iq3_xxs: CUDA dequantize works

  Kawrakow

  committed
  bf9349c

  Copy full SHA for bf9349c

• iq2_xxs: tuning quantization

  Kawrakow

  committed
  90faca2

  Copy full SHA for 90faca2

• iq3_xxs: starting to look better

  Show description for f120672
  Kawrakow

  committed
  f120672

  Copy full SHA for f120672

• iq3_xxs: CUDA dot product

  Show description for f1875b0
  Kawrakow

  committed
  f1875b0

  Copy full SHA for f1875b0

• iq3_xxs: scalar and AVX2 dot products

  Kawrakow

  committed
  c3b2029

  Copy full SHA for c3b2029

• iq3_xxs: ARM_NEON and Metal

  Show description for 1549302
  Kawrakow

  committed
  1549302

  Copy full SHA for 1549302

Commits on Jan 28, 2024

• iq3_xxs: slightly better grid points

  Kawrakow

  committed
  51cde19

  Copy full SHA for 51cde19

• Faster iq3_xxs and iq2_xs dot products on CUDA

  Kawrakow

  committed
  68cfcd4

  Copy full SHA for 68cfcd4

Commits on Jan 29, 2024

• iq3_xxs: add some quant mix

  Kawrakow

  committed
  7e4e748

  Copy full SHA for 7e4e748

• iq3_xxs: fix failing quantization test

  Show description for 6efbc69
  Kawrakow

  committed
  6efbc69

  Copy full SHA for 6efbc69

• iq3_xxs: hopefully fix ROCm

  Kawrakow

  committed
  6262343

  Copy full SHA for 6262343

• iq3_xxs: failing tests

  Show description for fe2160e
  Kawrakow

  committed
  fe2160e

  Copy full SHA for fe2160e

Commits on Jan 30, 2024

• Add IQ3_XXS to test-backend-ops

  Kawrakow

  committed
  fb6576b

  Copy full SHA for fb6576b