Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Automatically configure KV cache size #6

Merged
merged 17 commits into from
Mar 12, 2023
Merged

Automatically configure KV cache size #6

merged 17 commits into from
Mar 12, 2023

Conversation

WoosukKwon
Copy link
Collaborator

@WoosukKwon WoosukKwon commented Mar 3, 2023 •
edited
Loading

This PR adds OPT memory analyzer to the system, and uses it to automatically determine the KV cache size.

Tested models:

  • OPT-125M
  • OPT-350M
  • OPT-1.3B
  • OPT-2.7B
  • OPT-6.7B
  • OPT-13B

Tested GPUs:

  • A100

@WoosukKwon WoosukKwon merged commit e9d3f2f into main Mar 12, 2023
@WoosukKwon WoosukKwon deleted the autoconfig branch March 12, 2023 07:23
@TheBloke TheBloke mentioned this pull request Jul 20, 2023
Closed
xiangyuT added a commit to xiangyuT/vllm that referenced this pull request Oct 24, 2023
@gc-fu @xiangyuT
Support to run batched inference in underlying changes (vllm-project#6)
f78c169 
* finish changing scheduler

* finish merge

* fix model

* Fix (vllm-project#5)

* fix problems

* fix

* delete unused params

* remove redundant comments

---------

Co-authored-by: Xiangyu Tian <[email protected]>
@shanshanpt shanshanpt mentioned this pull request Nov 17, 2023
Closed
@junior-zsy junior-zsy mentioned this pull request Nov 20, 2023
Closed
hongxiayang pushed a commit to hongxiayang/vllm that referenced this pull request Feb 13, 2024
@WoosukKwon
Add memory analyzer & utomatically configure KV cache size (vllm-proj…
de10960 
…ect#6)
slyalin pushed a commit to slyalin/vllm that referenced this pull request Mar 21, 2024
@ilya-lavrenov
Merge pull request vllm-project#6 from mzegla/extended_requirements
2922b06 
Add missing Python requirements
mzusman added a commit to mzusman/vllm that referenced this pull request Apr 16, 2024
@mzusman
dtype (vllm-project#6)
00bce1f 
Co-authored-by: Mor Zusman <[email protected]>
dtrifiro referenced this pull request in dtrifiro/vllm Apr 26, 2024
@z103cb
Merge pull request #6 from z103cb/ibm_main_docker_ubi_updates
91e4a51 
[CI/Build] Dockerfile.ubi : Remove test stage
@dlopes78 dlopes78 mentioned this pull request May 8, 2024
Closed
Starmys pushed a commit to Starmys/vllm that referenced this pull request May 20, 2024
@xiaoxiawu-microsoft
Merge pull request vllm-project#6 from wenxcs/wenxh/fp8-on-a100
4e56e27 
FP8 on A100 for PHIMOE
@oliver-li oliver-li mentioned this pull request Jul 5, 2024
Closed
This was referenced Jul 5, 2024
Merged
Closed
@ehuaa ehuaa mentioned this pull request Jul 19, 2024
Closed
@xinzaifeixiang1992 xinzaifeixiang1992 mentioned this pull request Jul 24, 2024
Closed
@alixiaodi alixiaodi mentioned this pull request Aug 2, 2024
Closed
@Minami-su Minami-su mentioned this pull request Aug 11, 2024
Closed
@wangwensuo wangwensuo mentioned this pull request Aug 22, 2024
Closed
zeroorhero pushed a commit to zeroorhero/vllm that referenced this pull request Sep 23, 2024
@YaoJiayi
Merge pull request vllm-project#6 from KuntaiDu/kuntai-disagg-refactor
9eefec2 
Kuntai disagg refactor
@liulisi16323 liulisi16323 mentioned this pull request Sep 24, 2024
Closed
1 task
heheda12345 added a commit to heheda12345/vllm that referenced this pull request Sep 25, 2024
@heheda12345
Merge pull request vllm-project#6 from vllm-project/chang/num_prompt_…
9b931bf 
…tokens

[Bugfix] Include encoder_prompt_tokens in num_prompt_tokensin UsageInfo
@Clint-chan Clint-chan mentioned this pull request Sep 29, 2024
Open
1 task
@SpaceHunterInf SpaceHunterInf mentioned this pull request Sep 30, 2024
Closed
1 task
This was referenced Oct 12, 2024
Closed
Open
@alexchenyu alexchenyu mentioned this pull request Oct 13, 2024
Open
1 task
@xxzhang0927 xxzhang0927 mentioned this pull request Oct 30, 2024
Open
1 task
@hteeyeoh hteeyeoh mentioned this pull request Dec 6, 2024
Open
1 task
@HelenaSak HelenaSak mentioned this pull request Feb 19, 2025
Open
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@WoosukKwon