Code

Merge upstream nov11#59

Code

Merged

masahi merged 117 commits intooctoml:batch-servingoctoml/mlc-llm:batch-servingfrom masahi:merge-upstream-nov11masahi/mlc-llm:merge-upstream-nov11Copy head branch name to clipboardNov 9, 2023

+1,262-93

Commits

Commits on Oct 8, 2023

• [API] Add GenerationConfig (mlc-ai#1024)

  davidpissarra

  authored
  898db76

  Copy full SHA for 898db76

• Fix two bugs in kv-cache backtrack loop (mlc-ai#856)

  Show description for ad3a6b9
  shenberg

  authored
  ad3a6b9

  Copy full SHA for ad3a6b9

• [Build] Added --pdb flag to build.py, drop into pdb on error (mlc-ai#1017)

  Show description for 6e40c21
  Lunderberg

  authored
  6e40c21

  Copy full SHA for 6e40c21

• [Android] Use AlertDialog instead of Toast (mlc-ai#1039)

  cyx-6

  authored
  bae37b3

  Copy full SHA for bae37b3

Commits on Oct 9, 2023

• Add doc for ChatConfig, ConvConfig, GenerationConfig, BuildArgs (mlc-ai#1040)

  Show description for b44f679
  CharlieFRuan

  authored
  b44f679

  Copy full SHA for b44f679

• [Android] Add Llama2 q4f16_0 (mlc-ai#1041)

  Show description for 3a9849a
  spectrometerHBH

  authored
  3a9849a

  Copy full SHA for 3a9849a

• [Docs] Model prebuilts tracking page revamp (mlc-ai#1000)

  CharlieFRuan

  authored
  bed9e60

  Copy full SHA for bed9e60

• Update compile_models.rst (mlc-ai#1038)

  Show description for c02fdaf
  yongjer

  authored
  c02fdaf

  Copy full SHA for c02fdaf

• Support for the Stable LM 3B model (mlc-ai#1008)

  Show description for 85001ed
  jeethu

  authored
  85001ed

  Copy full SHA for 85001ed

• [Docs] Iterate model prebuilts docs (mlc-ai#1043)

  Show description for a032d40
  CharlieFRuan

  authored
  a032d40

  Copy full SHA for a032d40

• Update README.md

  junrushao

  authored
  a58605f

  Copy full SHA for a58605f

• [CPP] Separate common utils out from llm_chat.cc (mlc-ai#1044)

  Show description for bdd9d9b
  MasterJH5574

  authored
  bdd9d9b

  Copy full SHA for bdd9d9b

• Update README.md (mlc-ai#1045)

  Show description for 20131fb
  junrushao

  authored
  20131fb

  Copy full SHA for 20131fb

Commits on Oct 11, 2023

• add verbose stats to mlc-chat REST API (mlc-ai#1049)

  Show description for 1e6fb11
  denise-k

  authored
  1e6fb11

  Copy full SHA for 1e6fb11

Commits on Oct 12, 2023

• [Transform] Apply split_rotary optimization on prefill (mlc-ai#1033)

  Show description for b9179cf
  Lunderberg

  authored
  b9179cf

  Copy full SHA for b9179cf

• [Docs] Add mlc.ai/package to DEPENDENCY INSTALLATION group (mlc-ai#1055)

  Show description for 98ebd28
  

  LeshengJin

  and

  junrushao

  authored
  98ebd28

  Copy full SHA for 98ebd28

• Revert "[Transform] Apply split_rotary optimization on prefill (mlc-ai#1033)" (mlc-ai#1058)

  Show description for bfaa5b9
  MasterJH5574

  authored
  bfaa5b9

  Copy full SHA for bfaa5b9

Commits on Oct 13, 2023

• [BugFix] Set the right max_sequence_length for both Llama-1 and Llama-2 families (mlc-ai#1032)

  Show description for ca8c11b
  

  sunggg

  and

  “Sunghyun

  authored
  ca8c11b

  Copy full SHA for ca8c11b

• [Doc] Use -U instead of --force-reinstall (mlc-ai#1062)

  Show description for edab9b5
  junrushao

  authored
  edab9b5

  Copy full SHA for edab9b5

Commits on Oct 14, 2023

• [Model] Initial batching support for Llama (mlc-ai#1048)

  Show description for d854105
  MasterJH5574

  authored
  d854105

  Copy full SHA for d854105

• Fix Stable LM 3B build (mlc-ai#1061)

  Show description for c2b8cbc
  jeethu

  authored
  c2b8cbc

  Copy full SHA for c2b8cbc

• [Core] Remove duplication in MODEL.get_model calls (mlc-ai#1054)

  Show description for 481cd92
  Lunderberg

  authored
  481cd92

  Copy full SHA for 481cd92

• [ParamManager] Cleanup creation of quantization IRModule (mlc-ai#1053)

  Show description for 8184431
  Lunderberg

  authored
  8184431

  Copy full SHA for 8184431

Commits on Oct 15, 2023

• Minor typo fix (mlc-ai#1064)

  jeethu

  authored
  9010d48

  Copy full SHA for 9010d48

• Add links to Python API Reference (mlc-ai#1068)

  junrushao

  authored
  b0bfc88

  Copy full SHA for b0bfc88

• [Fix] ChatModule incorrect temperature buffer shape (mlc-ai#1070)

  Show description for 204860b
  MasterJH5574

  authored
  204860b

  Copy full SHA for 204860b

Commits on Oct 16, 2023

• [ParamManager] Added progress bar for get_item/set_item (mlc-ai#1063)

  Lunderberg

  authored
  d202077

  Copy full SHA for d202077

• [Python] Extract common device str parse function in ChatModule (mlc-ai#1074)

  Show description for 9872c48
  MasterJH5574

  authored
  9872c48

  Copy full SHA for 9872c48

Commits on Oct 17, 2023

• [Bugfix] Compilation Error in q4f32_1 (mlc-ai#1078)

  Show description for 3aefd9f
  junrushao

  authored
  3aefd9f

  Copy full SHA for 3aefd9f

Commits on Oct 19, 2023

• Establish mlc_chat.compiler (mlc-ai#1082)

  Show description for 2625945
  junrushao

  authored
  2625945

  Copy full SHA for 2625945

• Update README.md for Multi-GPU (mlc-ai#1090)

  junrushao

  authored
  56a8004

  Copy full SHA for 56a8004

• Support lib_path override in C++. Improvements on docs and error messages (mlc-ai#1086)

  Show description for b0373d1
  rickzx

  authored
  b0373d1

  Copy full SHA for b0373d1

• StreamIterator (mlc-ai#1057)

  Show description for 830656f
  

  varshith15

  and

  Varshith

  authored
  830656f

  Copy full SHA for 830656f

• Update benchmark.py according to mlc-ai#1086 (mlc-ai#1091)

  Show description for 9bf5723
  junrushao

  authored
  9bf5723

  Copy full SHA for 9bf5723

Commits on Oct 20, 2023

• Disable Disco for q4f16_ft and q8f16_ft quantization (mlc-ai#1094)

  LeshengJin

  authored
  62d0c03

  Copy full SHA for 62d0c03

• [Format] Apply isort and black for python/ (mlc-ai#1097)

  Show description for cf39bf6
  junrushao

  authored
  cf39bf6

  Copy full SHA for cf39bf6

Commits on Oct 21, 2023

• More formatting (mlc-ai#1099)

  junrushao

  authored
  e9b85ce

  Copy full SHA for e9b85ce

• Enable Python Linter (mlc-ai#1098)

  Show description for 03c641a
  junrushao

  authored
  03c641a

  Copy full SHA for 03c641a

• Add Basic Pylint and Mypy Tooling (mlc-ai#1100)

  Show description for 46d11e6
  junrushao

  authored
  46d11e6

  Copy full SHA for 46d11e6

Commits on Oct 22, 2023

• [CI] Add clang-format (mlc-ai#1103)

  junrushao

  authored
  6159cc4

  Copy full SHA for 6159cc4

Commits on Oct 23, 2023

• [Slim-LM] Smart path finding for config and weight (mlc-ai#1088)

  LeshengJin

  authored
  16dd2ae

  Copy full SHA for 16dd2ae

• [Transform] Provide IRModule transform for rewrite_attention (mlc-ai#1052)

  Show description for f57c9c9
  Lunderberg

  authored
  f57c9c9

  Copy full SHA for f57c9c9

• [ParamManager] Use BundleModelParams for transform_dequantize (mlc-ai#1056)

  Show description for e5927ce
  Lunderberg

  authored
  e5927ce

  Copy full SHA for e5927ce

• [Slim-LM] Introduce HFLoad for loading Pytorch and SafeTensor weights (mlc-ai#1113)

  LeshengJin

  authored
  7ae8c6d

  Copy full SHA for 7ae8c6d

Commits on Oct 24, 2023

• [WINDOWS] reduce noise in windows build (mlc-ai#1115)

  tqchen

  authored
  5a7dcd8

  Copy full SHA for 5a7dcd8

• Add CLI commands for compilation (mlc-ai#1109)

  junrushao

  authored
  61179a0

  Copy full SHA for 61179a0

• Auto updated submodule references

  

  Git bot

  committed
  8ce7793

  Copy full SHA for 8ce7793

• fix mismatched argument name (mlc-ai#1117)

  Show description for 488017d
  Sing-Li

  authored
  488017d

  Copy full SHA for 488017d

• [Docs] Add doc for max and mean gen len, shift factor; and buildArgs (mlc-ai#1119)

  Show description for 206103b
  CharlieFRuan

  authored
  206103b

  Copy full SHA for 206103b

• Revert "[ParamManager] Use BundleModelParams for transform_dequantize" (mlc-ai#1120)

  Show description for 2aa6809
  junrushao

  authored
  2aa6809

  Copy full SHA for 2aa6809

• Remove inaccurate warning message (mlc-ai#1121)

  Show description for 9cb8e8e
  junrushao

  authored
  9cb8e8e

  Copy full SHA for 9cb8e8e

• [REST] OpenAI compatible Rest API (mlc-ai#1107)

  Show description for 9166edb
  Kartik14

  authored
  9166edb

  Copy full SHA for 9166edb

Commits on Oct 25, 2023

• Add --opt flag parsing to CLI (mlc-ai#1123)

  junrushao

  authored
  a4279e3

  Copy full SHA for a4279e3

• [ParamManager][Redo] Use BundleModelParams for transform_dequantize (mlc-ai#1127)

  Show description for 973f9fc
  Lunderberg

  authored
  973f9fc

  Copy full SHA for 973f9fc

Commits on Oct 27, 2023

• added details to windows installation (mlc-ai#1133)

  Show description for 24f795e
  goutham2688

  authored
  24f795e

  Copy full SHA for 24f795e

Commits on Oct 28, 2023

• Grammatical and Typographical improvements (mlc-ai#1139)

  Show description for 2c492e5
  tmsagarofficial

  authored
  2c492e5

  Copy full SHA for 2c492e5

• Minor enhancements to ChatModule (mlc-ai#1132)

  Show description for 2ec0cc8
  YuchenJin

  authored
  2ec0cc8

  Copy full SHA for 2ec0cc8

Commits on Oct 29, 2023

• Updating tvm install docs (mlc-ai#1143)

  Show description for 27ac5ac
  David-Sharma

  authored
  27ac5ac

  Copy full SHA for 27ac5ac

• Make the help info consistent with program name (mlc-ai#1137)

  Show description for 2b6d832
  fennecJ

  authored
  2b6d832

  Copy full SHA for 2b6d832

• Support parameter packing (mlc-ai#1146)

  junrushao

  authored
  878ae84

  Copy full SHA for 878ae84

• [Slim-LM] Enable Group Quant (mlc-ai#1129)

  Show description for c0c3a8d
  zxybazh

  authored
  c0c3a8d

  Copy full SHA for c0c3a8d

• Enable Mypy and Pylint in mlc_chat Python Package (mlc-ai#1149)

  junrushao

  authored
  2193767

  Copy full SHA for 2193767

Commits on Oct 30, 2023

• Migrate Compiler Passes (mlc-ai#1150)

  junrushao

  authored
  0a25374

  Copy full SHA for 0a25374

• Compile Model Preset without External config.json (mlc-ai#1151)

  Show description for 1a79a53
  junrushao

  authored
  1a79a53

  Copy full SHA for 1a79a53

• Update attention layer (mlc-ai#1153)

  Show description for ba67835
  junrushao

  authored
  ba67835

  Copy full SHA for ba67835

• Add batched Llama model definition using vLLM paged attention (mlc-ai#1134)

  Show description for fee2cb5
  masahi

  authored
  fee2cb5

  Copy full SHA for fee2cb5

• [Transform][Redo] Apply split_rotary optimization on prefill (mlc-ai#1125)

  Show description for ece97b1
  Lunderberg

  authored
  ece97b1

  Copy full SHA for ece97b1

• Apply rewrite for normal attention and MQA (mlc-ai#1138)

  Show description for b190578
  Lunderberg

  authored
  b190578

  Copy full SHA for b190578

• [Rest] Fix emoji handling in Rest API. (mlc-ai#1142)

  YuchenJin

  authored
  8ca0176

  Copy full SHA for 8ca0176

• [Utility] Check for isinstance(exc, Exception) before entering pdb (mlc-ai#1095)

  Show description for 3cf5605
  Lunderberg

  authored
  3cf5605

  Copy full SHA for 3cf5605

• [Utils] Remove conversion to numpy array in utils.save_params (mlc-ai#1083)

  Show description for 0a9d6c7
  Lunderberg

  authored
  0a9d6c7

  Copy full SHA for 0a9d6c7

• [Fix][REST] Use lowered-cased "app" (mlc-ai#1159)

  junrushao

  authored
  425a2cb

  Copy full SHA for 425a2cb

Commits on Oct 31, 2023

• [Rest] Document emoji handling (mlc-ai#1160)

  Show description for 9076d01
  YuchenJin

  authored
  9076d01

  Copy full SHA for 9076d01

• Enable group quant transform with nn.Module (mlc-ai#1154)

  Show description for b5bfa5b
  

  cyx-6

  and

  junrushao

  authored
  b5bfa5b

  Copy full SHA for b5bfa5b

• Misc Cleanups of Compilation Pipeline (mlc-ai#1165)

  junrushao

  authored
  8438b27

  Copy full SHA for 8438b27

• Support CUDA Multi-Arch Compilation (mlc-ai#1166)

  junrushao

  authored
  02d1e57

  Copy full SHA for 02d1e57

• [Bugfix] Cannot find global function mlc.llm_chat_create (mlc-ai#1167)

  junrushao

  authored
  e0cd3f6

  Copy full SHA for e0cd3f6

Commits on Nov 1, 2023

• Fix RWKV Support (mlc-ai#1136)

  Show description for f5b2e88
  BBuf

  authored
  f5b2e88

  Copy full SHA for f5b2e88

• Auto updated submodule references

  

  Git bot

  committed
  200653a

  Copy full SHA for 200653a

• Fix Android app Permission denied error on Android 10 (mlc-ai#1175)

  Show description for 9831135
  

  anibohara2000

  and

  Animesh Bohara

  authored
  9831135

  Copy full SHA for 9831135

• [SLM] Fix group quantization (mlc-ai#1172)

  Show description for 1757777
  cyx-6

  authored
  1757777

  Copy full SHA for 1757777

Commits on Nov 2, 2023

• [Fix] TIR block name of dequantization (mlc-ai#1177)

  junrushao

  authored
  2ca7d15

  Copy full SHA for 2ca7d15

• [SLM][AutoLLM] Enable Command Line Weight Conversion (mlc-ai#1170)

  Show description for 53060af
  zxybazh

  authored
  53060af

  Copy full SHA for 53060af

Commits on Nov 3, 2023

• [Fix][SLM] Update q4f16 quantization with the new mutator name rule (mlc-ai#1178)

  Show description for 2dc8183
  LeshengJin

  authored
  2dc8183

  Copy full SHA for 2dc8183

• [Model Support][SWA] Add support for sliding window attention for Mistral (mlc-ai#1087)

  Show description for 6ae02dd
  

  CharlieFRuan

  and

  davidpissarra

  authored
  6ae02dd

  Copy full SHA for 6ae02dd

Commits on Nov 4, 2023

• Add Python API for Weight Conversion (mlc-ai#1182)

  Show description for 4716704
  junrushao

  authored
  4716704

  Copy full SHA for 4716704

• Merge llama_config.CONFIG into MODEL_PRESETS (mlc-ai#1188)

  junrushao

  authored
  9d20575

  Copy full SHA for 9d20575

• Merge llama_config.py into llama_model.py (mlc-ai#1189)

  junrushao

  authored
  5d1dc34

  Copy full SHA for 5d1dc34

• Add CodeLlama as part of model presets (mlc-ai#1190)

  junrushao

  authored
  4832c2f

  Copy full SHA for 4832c2f

• [Docs] Clarify zstd installation on Windows (mlc-ai#1191)

  junrushao

  authored
  78424f0

  Copy full SHA for 78424f0

• [Docs] Clarify zstd installation on Windows (mlc-ai#1196)

  Show description for 5d63f7e
  junrushao

  authored
  5d63f7e

  Copy full SHA for 5d63f7e

Commits on Nov 5, 2023

• Support overriding --max-sequence-length in command line (mlc-ai#1197)

  junrushao

  authored
  3417505

  Copy full SHA for 3417505

• [RestAPI] Added docs (mlc-ai#1193)

  Show description for 0e08845
  

  anibohara2000

  and

  Animesh Bohara

  authored
  0e08845

  Copy full SHA for 0e08845

• [API] llm-vscode extension support (mlc-ai#1198)

  Show description for 145a984
  davidpissarra

  authored
  145a984

  Copy full SHA for 145a984

• [Fix] Use fabs as floating point abs function in C++ (mlc-ai#1202)

  junrushao

  authored
  3413d17

  Copy full SHA for 3413d17

Commits on Nov 6, 2023

• Integrating MLC runtime with the new compilation workflow (mlc-ai#1203)

  junrushao

  authored
  7ccb51a

  Copy full SHA for 7ccb51a

• [Fix] Remove Redundant Warnings (mlc-ai#1204)

  Show description for 65478c8
  junrushao

  authored
  65478c8

  Copy full SHA for 65478c8

• Try fix macOS build with picojson (mlc-ai#1206)

  Show description for 01d4339
  junrushao

  authored
  01d4339

  Copy full SHA for 01d4339

• Try fix macOS build with picojson again (mlc-ai#1207)

  Show description for 51d6f9c
  junrushao

  authored
  51d6f9c

  Copy full SHA for 51d6f9c

• Auto updated submodule references

  

  Git bot

  committed
  a7f1183

  Copy full SHA for a7f1183

• [Fix] Keep update-to-date with upstream API change (mlc-ai#1209)

  junrushao

  authored
  e2c99a8

  Copy full SHA for e2c99a8

• Detect mtriple via LLVM (mlc-ai#1211)

  junrushao

  authored
  e00220c

  Copy full SHA for e00220c

• Fix Python3.8 compatibility breakage (mlc-ai#1210)

  Show description for 9869ca6
  Lunderberg

  authored
  9869ca6

  Copy full SHA for 9869ca6

• [Slim-LM] Enable loading from AWQ pre-quantized weight. (mlc-ai#1114)

  Show description for 4042626
  LeshengJin

  authored
  4042626

  Copy full SHA for 4042626

Commits on Nov 7, 2023

• [Bugfix] Fix Cannot import name '_LIB' from 'mlc_chat.base' (mlc-ai#1214)

  Show description for be1c18b
  CharlieFRuan

  authored
  be1c18b

  Copy full SHA for be1c18b

Commits on Nov 8, 2023

• [SLM] Support q3f16_1 and q4f32_1 (mlc-ai#1215)

  Show description for 1015aae
  cyx-6

  authored
  1015aae

  Copy full SHA for 1015aae

• Make the Compilation Working E2E (mlc-ai#1218)

  junrushao

  authored
  1a6fadd

  Copy full SHA for 1a6fadd

• [Mistral][SWA] Add sliding window to metadata (mlc-ai#1217)

  Show description for 616ca42
  CharlieFRuan

  authored
  616ca42

  Copy full SHA for 616ca42

• Support for chatml format conversation (for TinyLlama-1.1B-Chat-v0.2) (mlc-ai#956)

  Show description for e52f449
  acalatrava

  authored
  e52f449

  Copy full SHA for e52f449

• Add Rust Support for MLC-LLM (mlc-ai#1213)

  Show description for fbe75e3
  YuchenJin

  authored
  fbe75e3

  Copy full SHA for fbe75e3

• [Bugfix] Remove dependency on openai_api in chat module (mlc-ai#1222)

  Show description for beca2ab
  CharlieFRuan

  authored
  beca2ab

  Copy full SHA for beca2ab

• Bake in RAM Usage in the Generated DSO (mlc-ai#1224)

  Show description for 9ee5705
  junrushao

  authored
  9ee5705

  Copy full SHA for 9ee5705

• [Fix] ChatModule python messages and offset types (mlc-ai#1220)

  Show description for 069181c
  YuchenJin

  authored
  069181c

  Copy full SHA for 069181c

• [Fix] Variable Upperbound Should be Injected before Build Pipeline (mlc-ai#1225)

  Show description for f1bc951
  junrushao

  authored
  f1bc951

  Copy full SHA for f1bc951

Commits on Nov 9, 2023

• [MultiGPU] Support pre-sharded model weights (mlc-ai#1096)

  Show description for 834811f
  Lunderberg

  authored
  834811f

  Copy full SHA for 834811f

• Merge remote-tracking branch 'mlc-ai/main' into merge-upstream-nov11

  masahi

  committed
  d41ad34

  Copy full SHA for d41ad34

• fix

  masahi

  committed
  b022dc2

  Copy full SHA for b022dc2