[Bug]: 我在使用factory_llama工具以qlora的方式训练Qwen/Qwen2.5-1.5B-Instruct模型，然后以vllm加载lora的方式启动，结果报错：AttributeError: Model Qwen2ForCausalLM does not support BitsAndBytes quantization yet.，有大佬知道是哪儿的问题吗

[gaojuntian]

Your current environment

"""
This example shows how to use LoRA with different quantization techniques
for offline inference.

Requires HuggingFace credentials for access.
"""

import gc
from typing import List, Optional, Tuple

import torch
from huggingface_hub import snapshot_download

from vllm import EngineArgs, LLMEngine, RequestOutput, SamplingParams
from vllm.lora.request import LoRARequest

def create_test_prompts(
lora_path: str
) -> List[Tuple[str, SamplingParams, Optional[LoRARequest]]]:
return [
# this is an example of using quantization without LoRA
("My name is",
SamplingParams(temperature=0.0,
logprobs=1,
prompt_logprobs=1,
max_tokens=128), None),
# the next three examples use quantization with LoRA
("my name is",
SamplingParams(temperature=0.0,
logprobs=1,
prompt_logprobs=1,
max_tokens=128),
LoRARequest("lora-test-1", 1, lora_path)),
("The capital of USA is",
SamplingParams(temperature=0.0,
logprobs=1,
prompt_logprobs=1,
max_tokens=128),
LoRARequest("lora-test-2", 1, lora_path)),
("The capital of France is",
SamplingParams(temperature=0.0,
logprobs=1,
prompt_logprobs=1,
max_tokens=128),
LoRARequest("lora-test-3", 1, lora_path)),
]

def process_requests(engine: LLMEngine,
test_prompts: List[Tuple[str, SamplingParams,
Optional[LoRARequest]]]):
"""Continuously process a list of prompts and handle the outputs."""
request_id = 0

while test_prompts or engine.has_unfinished_requests():
    if test_prompts:
        prompt, sampling_params, lora_request = test_prompts.pop(0)
        engine.add_request(str(request_id),
                           prompt,
                           sampling_params,
                           lora_request=lora_request)
        request_id += 1

    request_outputs: List[RequestOutput] = engine.step()
    for request_output in request_outputs:
        if request_output.finished:
            print("----------------------------------------------------")
            print(f"Prompt: {request_output.prompt}")
            print(f"Output: {request_output.outputs[0].text}")

def initialize_engine(model: str, quantization: str,
lora_repo: Optional[str]) -> LLMEngine:
"""Initialize the LLMEngine."""

if quantization == "bitsandbytes":
    # QLoRA (https://arxiv.org/abs/2305.14314) is a quantization technique.
    # It quantizes the model when loading, with some config info from the
    # LoRA adapter repo. So need to set the parameter of load_format and
    # qlora_adapter_name_or_path as below.
    engine_args = EngineArgs(model=model,
                             quantization=quantization,
                             qlora_adapter_name_or_path=lora_repo,
                             load_format="bitsandbytes",
                             enable_lora=True,
                             max_lora_rank=64)
else:
    engine_args = EngineArgs(model=model,
                             quantization=quantization,
                             enable_lora=True,
                             max_loras=4)
return LLMEngine.from_engine_args(engine_args)

def main():
"""Main function that sets up and runs the prompt processing."""

test_configs = [{
    "name": "qlora_inference_example",
    'model': "/data/gaojuntian/glm4/LLaMA-Factory/model_dir/Qwen/Qwen2___5-1___5B",
    'quantization': "bitsandbytes",
    'lora_repo': '/data/gaojuntian/llama_last/LLaMA-Factory/saves/qwen2.5_1.5_bnb/lora/sft'
}]

for test_config in test_configs:
    print(
        f"~~~~~~~~~~~~~~~~ Running: {test_config['name']} ~~~~~~~~~~~~~~~~"
    )
    engine = initialize_engine(test_config['model'],
                               test_config['quantization'],
                               test_config['lora_repo'])
    lora_path = snapshot_download(repo_id=test_config['lora_repo'])
    test_prompts = create_test_prompts(lora_path)
    process_requests(engine, test_prompts)

    # Clean up the GPU memory for the next test
    del engine
    gc.collect()
    torch.cuda.empty_cache()

if name == 'main':
main()

Model Input Dumps

No response

🐛 Describe the bug

[Bug]: 我在使用factory_llama工具以qlora的方式训练Qwen/Qwen2.5-1.5B-Instruct模型，然后以vllm加载lora的方式启动，结果报错：AttributeError: Model Qwen2ForCausalLM does not support BitsAndBytes quantization yet.，有大佬知道是哪儿的问题吗

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[jeejeelee]

This feature will be included in the upcoming release, see: #9467 and #9574. You can consider manually build the main branch to address your issue

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!