Use dtype from model config & Add Dolly V2 #63
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hongxiayang
pushed a commit
to hongxiayang/vllm
that referenced
this pull request
Feb 13, 2024
yukavio
pushed a commit
to yukavio/vllm
that referenced
this pull request
Jul 3, 2024
SUMMARY: * delete NOTICE.txt file TEST PLAN: none Co-authored-by: andy-neuma <[email protected]>
dllehr-amd
pushed a commit
to dllehr-amd/vllm
that referenced
this pull request
Jul 22, 2024
dtrifiro
pushed a commit
to dtrifiro/vllm
that referenced
this pull request
Jul 30, 2024
sync release with main @ v0.5.0.post1-99-g8720c92e
JHLEE17
pushed a commit
to JHLEE17/vllm
that referenced
this pull request
Aug 1, 2024
* Add more detailed event names to profiler * Add more profiler stats * separate prompt and decode batch utilization * Add more metrics * revert engine/metrics.py changes * un-singletonify (what a funny word) habana profiler * formatting * add batch block utilization metric * fix division by zero * fix batch_block_utilization formula * minor refactors
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds the
defaultoption todtype, which uses FP16 for FP16 and FP32 models and BF16 for BF16 models. While this option will be used by default, users can specify the data type if they want to use BF16 for FP32 models.In addition, the PR integrates Dolly V2, a recent LLM with the GPT-NeoX architecture. The model is trained and saved in BF16.