Merge QKV into one linear layer #15
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
WoosukKwon
reviewed
Mar 30, 2023
Merged
WoosukKwon
approved these changes
Apr 2, 2023
bigPYJ1151
added a commit
to bigPYJ1151/vllm
that referenced
this pull request
Sep 12, 2023
hongxiayang
pushed a commit
to hongxiayang/vllm
that referenced
this pull request
Feb 13, 2024
slyalin
pushed a commit
to slyalin/vllm
that referenced
this pull request
Mar 26, 2024
…no-model-executor-opt [CPU] Avoid copy result and force allocation
z103cb
referenced
this pull request
in z103cb/opendatahub_vllm
Apr 22, 2024
This PR updates our grpc_server to add TGIS-style logs similar to https://github.com/IBM/text-generation-inference/blob/main/router/src/grpc_server.rs#L504-L512 This also disables the vllm per-request logging so that we don't double-log each request The timing info collected here is pretty rough, it doesn't plumb into the LLMEngine, it just times the generators to get the total time spent in the engine. We could do better, but this is a start. Example logs: ``` INFO 04-09 21:51:01 logs.py:43] generate_stream{input=[b'This is the story of Obama ridin...'] prefix_id= input_chars=[70] params=sampling { } stopping { max_new_tokens: 200 min_new_tokens: 16 } response { } decoding { } tokenization_time=0.45ms queue_and_inference_time=1096.67ms time_per_token=5.48ms total_time=1097.12ms input_toks=16}: Streaming response generated 200 tokens before NOT_FINISHED, output 848 chars: b' California. The story is told i...' INFO 04-09 21:51:08 logs.py:43] generate{input=[b'Lorem ipsum dolor sit amet, cons...', b'foooood man where is it'] prefix_id= input_chars=[469] params=sampling { } stopping { max_new_tokens: 20 min_new_tokens: 16 } response { } decoding { } tokenization_time=2.03ms queue_and_inference_time=122.23ms time_per_token=6.11ms total_time=124.26ms input_toks=124}: Sub-request 0 from batch of 2 generated 20 tokens before MAX_TOKENS, output 25 chars: b'?\\n\\n<!--\\n<!--\\n<!--\\n<!--\\n<!' INFO 04-09 21:51:08 logs.py:43] generate{input=[b'Lorem ipsum dolor sit amet, cons...', b'foooood man where is it'] prefix_id= input_chars=[469] params=sampling { } stopping { max_new_tokens: 20 min_new_tokens: 16 } response { } decoding { } tokenization_time=2.07ms queue_and_inference_time=122.22ms time_per_token=6.11ms total_time=124.29ms input_toks=7}: Sub-request 1 from batch of 2 generated 20 tokens before MAX_TOKENS, output 70 chars: b"?\\nI don't know.\\nI don't know.\\nI ..." ``` --------- Signed-off-by: Joe Runde <[email protected]> Signed-off-by: Joe Runde <[email protected]>
fxmarty
pushed a commit
to fxmarty/vllm-public
that referenced
this pull request
May 31, 2024
Correctly calculating the same value for the required cache blocks num for all torchrun processes
ykim362
pushed a commit
to ykim362/vllm
that referenced
this pull request
Jun 17, 2024
…-wenxh/fp8-on-a100-v5-pr Revert "0612 kernel of FP8 on A100"
Closed
1 task
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
@WoosukKwon Feel free to merge this after your review.