Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Misc] Use scalar type to dispatch to different gptq_marlin kernels #7323

Merged

Conversation

LucasWilkinson
Copy link
Collaborator

@LucasWilkinson LucasWilkinson commented Aug 9, 2024

Use ScalarType instead of num_bits (in combination with has_zp) to preform the dispatching for gptq_marlin this sets the stage for folding fp8_marlin.cu into gptq_marlin.cu since now we can move dequant_8bit in fp8_marlin.cu as a dequant<T, vllm::kFE4M3fn.id()>(int q) specialization. I did not fold fp8_marlin.cu into gptq_marlin.cu in this PR to avoid excessive compile times for gptq_marlin.cu, but once #7317 is completed then this should be folded in.

In-order to support passing scalar type as a template parameter in C++17, it has to be serialized to something that can be passed as a template parameter. This per introduces the concept of serializing the type into a 64 bit int id (that can be passed as a parameter) alongside a deserialization routine (from_id if the template needs to access the traits of the type). If/when we move to make C++20 the lowest standard support this serialization/deserialization can be removed as C++20 introduces passing literal class types as template parameters (see: https://en.cppreference.com/w/cpp/language/template_parameters)

Copy link

github-actions bot commented Aug 9, 2024

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which consists a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of default ones by unblocking the steps in your fast-check build on Buildkite UI.

Once the PR is approved and ready to go, please make sure to run full CI as it is required to merge (or just use auto-merge).

To run full CI, you can do one of these:

  • Comment /ready on the PR
  • Add ready label to the PR
  • Enable auto-merge.

🚀

Copy link
Contributor

@ProExpertProg ProExpertProg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! A minor nit and potential bug

Copy link
Contributor

@bnellnm bnellnm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@LucasWilkinson LucasWilkinson force-pushed the lwilkinson/gptq-scalar-type-dispatch branch from 5e15b86 to cc5247c Compare August 9, 2024 22:47
@LucasWilkinson LucasWilkinson force-pushed the lwilkinson/gptq-scalar-type-dispatch branch from cc5247c to 68424c9 Compare August 12, 2024 03:50
@LucasWilkinson
Copy link
Collaborator Author

/ready

@github-actions github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label Aug 12, 2024
@LucasWilkinson LucasWilkinson changed the title [Misc] Use scalar type to dispatch to diferent gptq_marlin kernels [Misc] Use scalar type to dispatch to different gptq_marlin kernels Aug 12, 2024
@tlrmchlsmth tlrmchlsmth merged commit 6aa33cb into vllm-project:main Aug 12, 2024
52 checks passed
Alvant pushed a commit to compressa-ai/vllm that referenced this pull request Oct 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ready ONLY add when PR is ready to merge/full CI is needed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants