Code

Upstream merge 25 01 27#391

Code

Merged

gshtras merged 109 commits intomainROCm/vllm:mainfrom upstream_merge_25_01_27ROCm/vllm:upstream_merge_25_01_27Copy head branch name to clipboardJan 28, 2025

+6,363-1,987

Commits

Commits on Jan 20, 2025

• [Misc] Update CODEOWNERS (vllm-project#12229)

  ywang96

  authored
  7bd3630

  Copy full SHA for 7bd3630

• fix: update platform detection for M-series arm based MacBook processors (vllm-project#12227)

  Show description for af69a6a
  isikhi

  authored
  af69a6a

  Copy full SHA for af69a6a

Commits on Jan 21, 2025

• [misc] add cuda runtime version to usage data (vllm-project#12190)

  Show description for da75122
  

  youkaichao

  and

  ywang96

  authored
  da75122

  Copy full SHA for da75122

• [bugfix] catch xgrammar unsupported array constraints (vllm-project#12210)

  Show description for 06a760d
  Jason-CKY

  authored
  06a760d

  Copy full SHA for 06a760d

• [Kernel] optimize moe_align_block_size for cuda graph and large num_experts (e.g. DeepSeek-V3) (vllm-project#12222)

  Show description for 750f4ca
  

  3 peopleauthored
  750f4ca

  Copy full SHA for 750f4ca

• Add quantization and guided decoding CODEOWNERS (vllm-project#12228)

  Show description for ecf6781
  mgoin

  authored
  ecf6781

  Copy full SHA for ecf6781

• [AMD][Build] Porting dockerfiles from the ROCm/vllm fork (vllm-project#11777)

  Show description for d4b62d4
  gshtras

  authored
  d4b62d4

  Copy full SHA for d4b62d4

• [BugFix] Fix GGUF tp>1 when vocab_size is not divisible by 64 (vllm-project#12230)

  Show description for 5fe6bf2
  NickLucche

  authored
  5fe6bf2

  Copy full SHA for 5fe6bf2

• [ci/build] disable failed and flaky tests (vllm-project#12240)

  Show description for 2fc6944
  youkaichao

  authored
  2fc6944

  Copy full SHA for 2fc6944

• [Misc] Rename MultiModalInputsV2 -> MultiModalInputs (vllm-project#12244)

  Show description for 9691255
  DarkLight1337

  authored
  9691255

  Copy full SHA for 9691255

• [Misc]Add BNB quantization for PaliGemmaForConditionalGeneration (vllm-project#12237)

  Show description for 1f1542a
  jeejeelee

  authored
  1f1542a

  Copy full SHA for 1f1542a

• [Misc] Remove redundant TypeVar from base model (vllm-project#12248)

  Show description for f2e9f2a
  DarkLight1337

  authored
  f2e9f2a

  Copy full SHA for f2e9f2a

• [Bugfix] Fix mm_limits access for merged multi-modal processor (vllm-project#12252)

  Show description for a94eee4
  DarkLight1337

  authored
  a94eee4

  Copy full SHA for a94eee4

• [torch.compile] transparent compilation with more logging (vllm-project#12246)

  Show description for c81081f
  youkaichao

  authored
  c81081f

  Copy full SHA for c81081f

• [V1][Bugfix] Fix data item ordering in mixed-modality inference (vllm-project#12259)

  Show description for b197a5c
  ywang96

  authored
  b197a5c

  Copy full SHA for b197a5c

• Remove pytorch comments for outlines + compressed-tensors (vllm-project#12260)

  Show description for 9a7c3a0
  tdoublep

  authored
  9a7c3a0

  Copy full SHA for 9a7c3a0

• [Platform] improve platforms getattr (vllm-project#12264)

  Show description for c646128
  MengqingCao

  authored
  c646128

  Copy full SHA for c646128

• [ci/build] update nightly torch for gh200 test (vllm-project#12270)

  Show description for 3aec49e
  youkaichao

  authored
  3aec49e

  Copy full SHA for 3aec49e

• [Bugfix] fix race condition that leads to wrong order of token returned (vllm-project#10802)

  Show description for 9705b90
  joennlae

  authored
  9705b90

  Copy full SHA for 9705b90

• [Kernel] fix moe_align_block_size error condition (vllm-project#12239)

  Show description for 1e60f87
  jinzhen-lin

  authored
  1e60f87

  Copy full SHA for 1e60f87

• [v1][stats][1/n] Add RequestStatsUpdate and RequestStats types (vllm-project#10907)

  Show description for 132a132
  rickyyx

  authored
  132a132

  Copy full SHA for 132a132

• [Bugfix] Multi-sequence broken (vllm-project#11898)

  Show description for 18fd4a8
  andylolu2

  authored
  18fd4a8

  Copy full SHA for 18fd4a8

• [Misc] Remove experimental dep from tracing.py (vllm-project#12007)

  Show description for 347eeeb
  codefromthecrypt

  authored
  347eeeb

  Copy full SHA for 347eeeb

• [Misc] Set default backend to SDPA for get_vit_attn_backend (vllm-project#12235)

  Show description for fa9ee08
  wangxiyuan

  authored
  fa9ee08

  Copy full SHA for fa9ee08

• [Core] Free CPU pinned memory on environment cleanup (vllm-project#10477)

  janimo

  authored
  9c485d9

  Copy full SHA for 9c485d9

• [bugfix] moe tuning. rm is_navi() (vllm-project#12273)

  Show description for 2acba47
  divakar-amd

  authored
  2acba47

  Copy full SHA for 2acba47

• [BUGFIX] When skip_tokenize_init and multistep are set, execution crashes (vllm-project#12277)

  Show description for 69196a9
  

  maleksan85

  and

  maleksan85

  authored
  69196a9

  Copy full SHA for 69196a9

• [Documentation][AMD] Add information about prebuilt ROCm vLLM docker for perf validation purpose (vllm-project#12281)

  Show description for 09ccc9c
  hongxiayang

  authored
  09ccc9c

  Copy full SHA for 09ccc9c

Commits on Jan 22, 2025

• [VLM] Simplify post-processing of replacement info (vllm-project#12269)

  Show description for df76e5a
  DarkLight1337

  authored
  df76e5a

  Copy full SHA for df76e5a

• [ci/lint] Add back default arg for pre-commit (vllm-project#12279)

  Show description for 64ea24d
  khluu

  authored
  64ea24d

  Copy full SHA for 64ea24d

• [CI] add docker volume prune to neuron CI (vllm-project#12291)

  Show description for 016e367
  liangfu

  authored
  016e367

  Copy full SHA for 016e367

• [Ci/Build] Fix mypy errors on main (vllm-project#12296)

  Show description for cbdc4ad
  DarkLight1337

  authored
  cbdc4ad

  Copy full SHA for cbdc4ad

• [Benchmark] More accurate TPOT calc in benchmark_serving.py (vllm-project#12288)

  Show description for 222a9dc
  njhill

  authored
  222a9dc

  Copy full SHA for 222a9dc

• [core] separate builder init and builder prepare for each batch (vllm-project#12253)

  Show description for 66818e5
  youkaichao

  authored
  66818e5

  Copy full SHA for 66818e5

• [Build] update requirements of no-device (vllm-project#12299)

  Show description for 4004f14
  MengqingCao

  authored
  4004f14

  Copy full SHA for 4004f14

• [Core] Support fully transparent sleep mode (vllm-project#11743)

  Show description for 68ad4e3
  youkaichao

  authored
  68ad4e3

  Copy full SHA for 68ad4e3

• [VLM] Avoid unnecessary tokenization (vllm-project#12310)

  Show description for cd7b6f0
  DarkLight1337

  authored
  cd7b6f0

  Copy full SHA for cd7b6f0

• [Model][Bugfix]: correct Aria model output (vllm-project#12309)

  Show description for 528dbca
  xffxff

  authored
  528dbca

  Copy full SHA for 528dbca

• [Bugfix][VLM] Fix mixed-modality inference backward compatibility for V0 (vllm-project#12313)

  Show description for 16366ee
  ywang96

  authored
  16366ee

  Copy full SHA for 16366ee

• [Doc] Add docs for prompt replacement (vllm-project#12318)

  Show description for 6609cdf
  DarkLight1337

  authored
  6609cdf

  Copy full SHA for 6609cdf

• [Misc] Fix the error in the tip for the --lora-modules parameter (vllm-project#12319)

  Show description for fc66dee
  WangErXiao

  authored
  fc66dee

  Copy full SHA for fc66dee

• [Misc] Improve the readability of BNB error messages (vllm-project#12320)

  Show description for 84bee4b
  jeejeelee

  authored
  84bee4b

  Copy full SHA for 84bee4b

• [Bugfix] Fix HPU multiprocessing executor (vllm-project#12167)

  Show description for 96f6a75
  kzawora-intel

  authored
  96f6a75

  Copy full SHA for 96f6a75

• [Core] Support reset_prefix_cache (vllm-project#12284)

  comaniac

  authored
  7206ce4

  Copy full SHA for 7206ce4

• [Frontend][V1] Online serving performance improvements (vllm-project#12287)

  njhill

  authored
  aea9436

  Copy full SHA for aea9436

Commits on Jan 23, 2025

• [AMD][Quantization] Add TritonScaledMMLinearKernel since int8 is broken for AMD (vllm-project#12282)

  Show description for 68c4421
  rasmith

  authored
  68c4421

  Copy full SHA for 68c4421

• [Bugfix] Fixing AMD LoRA CI test. (vllm-project#12329)

  Show description for 8d7aa9d
  Alexei-V-Ivanov-AMD

  authored
  8d7aa9d

  Copy full SHA for 8d7aa9d

• [Docs] Update FP8 KV Cache documentation (vllm-project#12238)

  Show description for 01a5594
  

  mgoin

  and

  DarkLight1337

  authored
  01a5594

  Copy full SHA for 01a5594

• [Docs] Document vulnerability disclosure process (vllm-project#12326)

  Show description for 7551a34
  russellb

  authored
  7551a34

  Copy full SHA for 7551a34

• [V1] Add uncache_blocks (vllm-project#12333)

  comaniac

  authored
  f0ef372

  Copy full SHA for f0ef372

• [doc] explain common errors around torch.compile (vllm-project#12340)

  Show description for 5116274
  youkaichao

  authored
  5116274

  Copy full SHA for 5116274

• [Hardware][Gaudi][BugFix] Fix dataclass error due to triton package update (vllm-project#12338)

  Show description for 8ae5ff2
  zhenwei-intel

  authored
  8ae5ff2

  Copy full SHA for 8ae5ff2

• [Bugfix] Fix k_proj's bias for whisper self attention (vllm-project#12342)

  Show description for c5b4b11
  Isotr0py

  authored
  c5b4b11

  Copy full SHA for c5b4b11

• [Kernel] Flash Attention 3 Support (vllm-project#12093)

  Show description for 978b45f
  LucasWilkinson

  authored
  978b45f

  Copy full SHA for 978b45f

• [Doc] Troubleshooting errors during model inspection (vllm-project#12351)

  Show description for d07efb3
  DarkLight1337

  authored
  d07efb3

  Copy full SHA for d07efb3

• [V1] Simplify M-RoPE (vllm-project#12352)

  Show description for 99d01a5
  

  ywang96

  and

  imkero

  authored
  99d01a5

  Copy full SHA for 99d01a5

• [Bugfix] Fix broken internvl2 inference with v1 (vllm-project#12360)

  Show description for 8c01b80
  Isotr0py

  authored
  8c01b80

  Copy full SHA for 8c01b80

• [core] add wake_up doc and some sanity check (vllm-project#12361)

  Show description for 3f50c14
  youkaichao

  authored
  3f50c14

  Copy full SHA for 3f50c14

• [torch.compile] decouple compile sizes and cudagraph sizes (vllm-project#12243)

  Show description for 6e650f5
  youkaichao

  authored
  6e650f5

  Copy full SHA for 6e650f5

• [FP8][Kernel] Dynamic kv cache scaling factors computation (vllm-project#11906)

  Show description for e97f802
  

  gshtras

  and

  micah-wil

  authored
  e97f802

  Copy full SHA for e97f802

• [TPU] Update TPU CI to use torchxla nightly on 20250122 (vllm-project#12334)

  Show description for 2c85529
  lsy323

  authored
  2c85529

  Copy full SHA for 2c85529

• [Docs] Document Phi-4 support (vllm-project#12362)

  Show description for 2cbeeda
  Isotr0py

  authored
  2cbeeda

  Copy full SHA for 2cbeeda

• [BugFix] Fix parameter names and process_after_weight_loading for W4A16 MoE Group Act Order (vllm-project#11528)

  Show description for eb5cb5e
  

  3 peopleauthored
  eb5cb5e

  Copy full SHA for eb5cb5e

• [Misc] Fix OpenAI API Compatibility Issues in Benchmark Script (vllm-project#12357)

  Show description for 9726ad6
  jsato8094

  authored
  9726ad6

  Copy full SHA for 9726ad6

• [Docs] Add meetup slides (vllm-project#12345)

  Show description for 682b55b
  WoosukKwon

  authored
  682b55b

  Copy full SHA for 682b55b

Commits on Jan 24, 2025

• [Docs] Update spec decode + structured output in compat matrix (vllm-project#12373)

  Show description for c5cffcd
  russellb

  authored
  c5cffcd

  Copy full SHA for c5cffcd

• [V1][Frontend] Coalesce bunched RequestOutputs (vllm-project#12298)

  Show description for 24b0205
  

  njhill

  and

  robertgshaw2-redhat

  authored
  24b0205

  Copy full SHA for 24b0205

• Set weights_only=True when using torch.load() (vllm-project#12366)

  Show description for d3d6bb1
  russellb

  authored
  d3d6bb1

  Copy full SHA for d3d6bb1

• [Bugfix] Path join when building local path for S3 clone (vllm-project#12353)

  Show description for 5e5630a
  omer-dayan

  authored
  5e5630a

  Copy full SHA for 5e5630a

• Update compressed-tensors version (vllm-project#12367)

  dsikka

  authored
  55ef66e

  Copy full SHA for 55ef66e

• [V1] Increase default batch size for H100/H200 (vllm-project#12369)

  Show description for 0e74d79
  WoosukKwon

  authored
  0e74d79

  Copy full SHA for 0e74d79

• [perf] fix perf regression from vllm-project#12253 (vllm-project#12380)

  Show description for 6dd94db
  youkaichao

  authored
  6dd94db

  Copy full SHA for 6dd94db

• [Misc] Use VisionArena Dataset for VLM Benchmarking (vllm-project#12389)

  Show description for 3c818bd
  ywang96

  authored
  3c818bd

  Copy full SHA for 3c818bd

• [ci/build] fix wheel size check (vllm-project#12396)

  Show description for c7c9851
  youkaichao

  authored
  c7c9851

  Copy full SHA for c7c9851

• [Hardware][Gaudi][Doc] Add missing step in setup instructions (vllm-project#12382)

  MohitIntel

  authored
  9a0f3bd

  Copy full SHA for 9a0f3bd

• [ci/build] sync default value for wheel size (vllm-project#12398)

  Show description for e784c6b
  youkaichao

  authored
  e784c6b

  Copy full SHA for e784c6b

• [Misc] Enable proxy support in benchmark script (vllm-project#12356)

  Show description for 3bb8e2c
  jsato8094

  authored
  3bb8e2c

  Copy full SHA for 3bb8e2c

• [Bugfix][Kernel] Fix CUDA 11.8 being broken by FA3 build (vllm-project#12375)

  Show description for ab5bbf5
  LucasWilkinson

  authored
  ab5bbf5

  Copy full SHA for ab5bbf5

• [Misc] Remove deprecated code (vllm-project#12383)

  Show description for df5dafa
  DarkLight1337

  authored
  df5dafa

  Copy full SHA for df5dafa

• [Bugfix][Kernel] FA3 Fix - RuntimeError: This flash attention build only supports pack_gqa (for build size reasons). (vllm-project#12405)

  Show description for 3132a93
  LucasWilkinson

  authored
  3132a93

  Copy full SHA for 3132a93

Commits on Jan 25, 2025

• [Bugfix][Kernel] Fix moe align block issue for mixtral (vllm-project#12413)

  ElizaWszola

  authored
  221d388

  Copy full SHA for 221d388

• [Bugfix] Fix BLIP-2 processing (vllm-project#12412)

  Show description for fb30ee9
  DarkLight1337

  authored
  fb30ee9

  Copy full SHA for fb30ee9

• [ROCm][MoE] MI300 tuned configs Mixtral-8x(7B,22B) | fp16, fp8 (vllm-project#12408)

  Show description for bf21481
  divakar-amd

  authored
  bf21481

  Copy full SHA for bf21481

• [Misc] Add FA2 support to ViT MHA layer (vllm-project#12355)

  Show description for f1fc051
  Isotr0py

  authored
  f1fc051

  Copy full SHA for f1fc051

• [TPU][CI] Update torchxla version in requirement-tpu.txt (vllm-project#12422)

  Show description for 324960a
  lsy323

  authored
  324960a

  Copy full SHA for 324960a

Commits on Jan 26, 2025

• [Misc][Bugfix] FA3 support to ViT MHA layer (vllm-project#12435)

  Show description for 2a0309a
  

  ywang96

  and

  Isotr0py

  authored
  2a0309a

  Copy full SHA for 2a0309a

• [V1][Perf] Reduce scheduling overhead in model runner after cuda sync (vllm-project#12094)

  Show description for fa63e71
  youngkent

  authored
  fa63e71

  Copy full SHA for fa63e71

• [V1][Bugfix] Fix assertion when mm hashing is turned off (vllm-project#12439)

  Show description for 0ee349b
  ywang96

  authored
  0ee349b

  Copy full SHA for 0ee349b

• [Misc] Revert FA on ViT vllm-project#12355 and vllm-project#12435 (vllm-project#12445)

  ywang96

  authored
  a525527

  Copy full SHA for a525527

• [Frontend] generation_config.json for maximum tokens(vllm-project#12242)

  Show description for 9ddc352
  

  9 peopleauthored
  9ddc352

  Copy full SHA for 9ddc352

• [Bugfix] Disable w16a16 2of4 sparse CompressedTensors24 (vllm-project#12417)

  Show description for aa2cd2c
  

  tlrmchlsmth

  and

  mgoin

  authored
  aa2cd2c

  Copy full SHA for aa2cd2c

• [Bugfix/CI] Fix broken kernels/test_mha.py (vllm-project#12450)

  tlrmchlsmth

  authored
  72f4880

  Copy full SHA for 72f4880

• [Bugfix][Kernel] Fix perf regression caused by PR vllm-project#12405 (vllm-project#12434)

  Show description for 68f1114
  LucasWilkinson

  authored
  68f1114

  Copy full SHA for 68f1114

• [Build/CI] Fix libcuda.so linkage (vllm-project#12424)

  Show description for 72bac73
  tlrmchlsmth

  authored
  72bac73

  Copy full SHA for 72bac73

Commits on Jan 27, 2025

• [Frontend] Rerank API (Jina- and Cohere-compatible API) (vllm-project#12376)

  Show description for 0034b09
  K-Mistele

  authored
  0034b09

  Copy full SHA for 0034b09

• [DOC] Add link to vLLM blog (vllm-project#12460)

  Show description for 582cf78
  terrytangyuan

  authored
  582cf78

  Copy full SHA for 582cf78

• [V1] Avoid list creation in input preparation (vllm-project#12457)

  Show description for 28e0750
  WoosukKwon

  authored
  28e0750

  Copy full SHA for 28e0750

• [Frontend] Support scores endpoint in run_batch (vllm-project#12430)

  Show description for 0cc6b38
  pooyadavoodi

  authored
  0cc6b38

  Copy full SHA for 0cc6b38

• [Bugfix] Fix Granite 3.0 MoE model loading (vllm-project#12446)

  Show description for 5204ff5
  DarkLight1337

  authored
  5204ff5

  Copy full SHA for 5204ff5

• [Bugfix] Fix missing seq_start_loc in xformers prefill metadata (vllm-project#12464)

  Show description for 372bf08
  Isotr0py

  authored
  372bf08

  Copy full SHA for 372bf08

• [V1][Minor] Minor optimizations for update_from_output (vllm-project#12454)

  Show description for 624a1e4
  WoosukKwon

  authored
  624a1e4

  Copy full SHA for 624a1e4

• [Bugfix] Fix gpt2 GGUF inference (vllm-project#12467)

  Show description for ce69f7f
  Isotr0py

  authored
  ce69f7f

  Copy full SHA for ce69f7f

• [Build] Only build 9.0a for scaled_mm and sparse kernels (vllm-project#12339)

  Show description for 103bd17
  LucasWilkinson

  authored
  103bd17

  Copy full SHA for 103bd17

• [V1][Metrics] Add initial Prometheus logger (vllm-project#12416)

  Show description for 01ba927
  markmc

  authored
  01ba927

  Copy full SHA for 01ba927

• [V1][CI/Test] Do basic test for top-p & top-k sampling (vllm-project#12469)

  Show description for 3f1fc74
  WoosukKwon

  authored
  3f1fc74

  Copy full SHA for 3f1fc74

• [FlashInfer] Upgrade to 0.2.0 (vllm-project#11194)

  Show description for 2bc3fbb
  

  abmfy

  and

  youkaichao

  authored
  2bc3fbb

  Copy full SHA for 2bc3fbb

• Merge remote-tracking branch 'upstream/main'

  gshtras

  committed
  8e6d987

  Copy full SHA for 8e6d987

Commits on Jan 28, 2025

• Merge remote-tracking branch 'origin/main' into upstream_merge_25_01_27

  gshtras

  committed
  a892ecc

  Copy full SHA for a892ecc

• Direct call on ROCm

  gshtras

  committed
  c8b8654

  Copy full SHA for c8b8654