[Usage]: How to inference the quantized base model with LoRA weights ?

[thincal]

Your current environment

installed vLLM + GPU env.

How would you like to use vllm

I have done the QLoRA training with AWQ quantized base model, is that possible using the vLLM to load the AWQ base model and inference with the lora weights directly without merging ?

[jeejeelee]

The current vllm does not support QLoRA yet

[thincal]

is there any plan for the qlora inference ?

[jeejeelee]

@thincal perhaps this PR can help you #2828