Skip to content
This repository has been archived by the owner on Oct 11, 2024. It is now read-only.

upstream merge sync 2024-03-11 #108

Merged
merged 90 commits into from
Mar 12, 2024
Merged

upstream merge sync 2024-03-11 #108

merged 90 commits into from
Mar 12, 2024

Conversation

andy-neuma
Copy link
Member

@andy-neuma andy-neuma commented Mar 11, 2024 •
edited
Loading

SUMMARY:

  • upstream merge (sync) up to 657061fdced8a33a60c1b09f5da2525de9da8f03
  • some minor changes related to ruff and yapf

NOTES: we are now consistently getting out memory of errors when running tests/models/test_marlin.py. i've disabled the test and created an ASANA ticket to track the issue.

TEST PLAN:
runs on remote push

ronensc and others added 30 commits February 21, 2024 18:18
@ronensc
Update comment (vllm-project#2934)
d7f3964
@Maxusmusti
Added early stopping to completion APIs (vllm-project#2939)
5574081
@esmeetu
Migrate MistralForCausalLM to LlamaForCausalLM (vllm-project#2868)
344020c
@WoosukKwon
Use Llama RMSNorm custom op for Gemma (vllm-project#2974)
95529e3
@mspronesti
chore(vllm): codespell for spell checking (vllm-project#2820)
93dc5a2
@WoosukKwon
Optimize GeGLU layer in Gemma (vllm-project#2975)
fd5dcc5
@44670
[FIX] Fix a bug in initializing Yarn RoPE (vllm-project#2983)
c530e2c
@WoosukKwon
Remove Flash Attention in test env (vllm-project#2982)
6f32cdd
@ronensc
Include tokens from prompt phase in counter_generation_tokens (vllm…
4caf704 
…-project#2802)
@zhaoyang-star
Fix nvcc not found in vlm-openai image (vllm-project#2781)
57f0449
@WoosukKwon
[Fix] Fissertion on YaRN model len (vllm-project#2984)
f7c1234
@hmellor
Port metrics from aioprometheus to prometheus_client (vllm-projec…
ef978fe 
…t#2730)
@jlcmoore
Add LogProbs for Chat Completions in OpenAI (vllm-project#2918)
70f3e8e
@pcmoritz @cadedaniel
Optimize Triton MoE Kernel (vllm-project#2979)
cfc15a1 
Co-authored-by: Cade Daniel <[email protected]>
@WoosukKwon
[Minor] Remove gather_cached_kv kernel (vllm-project#3043)
d6e4a13
@esmeetu
[Minor] Remove unused config files (vllm-project#3039)
d9f726c
@esmeetu
Don't use cupy when enforce_eager=True (vllm-project#3037)
c1c0d00
@esmeetu
Fix stablelm (vllm-project#3038)
4dd6416
@dachengai @WoosukKwon
Support Orion model (vllm-project#2539)
48a8f4a 
Co-authored-by: zhangdacheng <[email protected]>
Co-authored-by: Woosuk Kwon <[email protected]>
@Jingru
fix get_ip error in pure ipv6 environment (vllm-project#2931)
2410e32
@WoosukKwon
[Minor] Fix type annotation in fused moe (vllm-project#3045)
4bd18ec
@dylanwhawk
Support logit bias for OpenAI API (vllm-project#3027)
e0ade06
@WoosukKwon
[Minor] Fix StableLMEpochForCausalLM -> StableLmForCausalLM (vllm-pro…
8b430d7 
…ject#3046)
@sighingnow
Enable GQA support in the prefix prefill kernels (vllm-project#3007)
71bcaf9 
Signed-off-by: Tao He <[email protected]>
@ElefHead
multi-lora documentation fix (vllm-project#3064)
a868310
@AllenDou
Restrict prometheus_client >= 0.18.0 to prevent errors when importing…
e46fa5d 
… pkgs (vllm-project#3070)
@liangfu
[Neuron] Support inference with transformers-neuronx (vllm-project#2569)
3b7178c
@WoosukKwon
Add LoRA support for Gemma (vllm-project#3050)
929b4f2
@chu-tianxiang
Add Support for 2/3/8-bit GPTQ Quantization Models (vllm-project#2330)
01a5d18
@jaywonchung
Fix: AttributeError in OpenAI-compatible server (vllm-project#3018)
a6d471c
WoosukKwon and others added 13 commits March 8, 2024 10:52
@WoosukKwon
[FIX] Make flash_attn optional (vllm-project#3269)
1cb0cc2
@mgoin
Move model filelocks from /tmp/ to ~/.cache/vllm/locks/ dir (vllm…
c2c5e09 
…-project#3241)
@zhuohan123
[FIX] Fix prefix test error on main (vllm-project#3286)
f48c679
@cadedaniel
[Speculative decoding 3/9] Worker which speculates, scores, and appli…
8437bae 
…es rejection sampling (vllm-project#3103)
@tterrysun
Enhance lora tests with more layer and rank variations (vllm-project#…
0bba88d 
…3243)
@dllehr-amd
[ROCM] Fix blockReduceSum to use correct warp counts for ROCm and CUDA (
e4a28e5 
vllm-project#3262)
@esmeetu
[BugFix] Fix get tokenizer when using ray (vllm-project#3301)
9e8744a
@njhill
[Fix] Fix best_of behavior when n=1 (vllm-project#3298)
4b59f00
@zhuohan123
Re-enable the 80 char line width limit (vllm-project#3305)
2f8844b
@pcmoritz
[docs] Add LoRA support information for models (vllm-project#3299)
657061f
Merge remote-tracking branch 'upstream/main' into upstream-sync-2024-…
46d67bf 
…03-11
yapf formatting
9179ac9
ruff and yapf patches
44eb021
@andy-neuma andy-neuma changed the title Upstream sync 2024 03 11 upstream merge sync 2024-03-11 Mar 11, 2024
robertgshaw2-redhat
robertgshaw2-redhat approved these changes Mar 11, 2024
View reviewed changes
Copy link
Collaborator

@robertgshaw2-redhat robertgshaw2-redhat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM pending automation

Main changes are:

  • testing
  • lora
  • speculative decoding
  • attn refactor

@andy-neuma andy-neuma merged commit 7bbd2cc into main Mar 12, 2024
2 checks passed
@andy-neuma andy-neuma deleted the upstream-sync-2024-03-11 branch March 12, 2024 11:22
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@robertgshaw2-redhat robertgshaw2-redhat robertgshaw2-redhat approved these changes

@tlrmchlsmth tlrmchlsmth Awaiting requested review from tlrmchlsmth

@mgoin mgoin Awaiting requested review from mgoin

@varun-sundar-rabindranath varun-sundar-rabindranath Awaiting requested review from varun-sundar-rabindranath

@bnellnm bnellnm Awaiting requested review from bnellnm

@SageMoore SageMoore Awaiting requested review from SageMoore

@ElizaWszola ElizaWszola Awaiting requested review from ElizaWszola

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.