[Bug]: cannot load model back due to [does not appear to have a file named config.json]

[yananchen1989]

Your current environment

The output of `python collect_env.py`

🐛 Describe the bug

hi there,

vllm version: 0.4.1

I fine-tuned the mistral-7b-v0.2 model using the trainer of huggingface https://huggingface.co/docs/trl/v0.8.6/trainer
the training worked well and finally it saved the model, as bellow:
path_to_the_model:

adapter_config.json
adapter_model.safetensors
checkpoint-16
checkpoint-24
checkpoint-8
README.md
special_tokens_map.json
tokenizer_config.json
tokenizer.json
tokenizer.model
training_args.bin

it can successfully be loaded back using AutoModelForCausalLM.from_pretrained
However, when I try to load it back via vllm, it caused error:

does not appear to have a file named config.json

The vllm codes:

from langchain_community.llms import VLLM
llm = VLLM(model="path_to_the_model", 
       trust_remote_code=True,  # mandatory for hf models
       max_new_tokens=64,
       temperature=0,
       # tensor_parallel_size=... # for distributed inference
)

from vllm import LLM, SamplingParams
from vllm.lora.request import LoRARequest
llm = LLM(model="path_to_the_model", enable_lora=True)

neither works.
any solutions ?

[PierreCarceller]

Having the same probleme here.

Modifing this script

Replacing TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF by bartowski/Meta-Llama-3.1-8B-Instruct-GGUF giving this error :

OSError: bartowski/Meta-Llama-3.1-8B-Instruct-GGUF does not appear to have a file named config.json. Checkout 'https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF/tree/main' for available files

[I321065]

same issue, +1

vllm version: v0.5.5
model: *.GGUF

for GGUF model, I do not think config.json is necessary, please tell me if I am wrong

[paolovic]

+1
vLLM v0.5.5
model: DeepSeek-Coder-V2-Instruct-Q4_K_M.gguf

OSError: /DeepSeek-Coder-V2-Instruct-GGUF does not appear to have a file named config.json. Checkout 'https://huggingface.co//u01/data/analytics/models/DeepSeek-Coder-V2-Instruct-GGUF/tree/None' for available files.

[bartowski1182]

Alternatively, if there's a way to know what should be added in a config.json, I'm happy to start adding them..

[PierreCarceller]

@bartowski1182 I tryed to add localy a config.json to this repo.

My config.json contained this:

{
     "model_type": "cohere",
}

I was inspired by what TheBloke was doing here

I got another error telling me that VLLm didn't support models of type "cohere". But the first mistake was fixed.

It might work with other repo's (aka LLama2)

[bartowski1182]

It would also be easier if it just expected the "ForCausalLM" format that is in existing config.jsons cause then I could just parse that and throw it in, but I've seen it being "llama" in theblokes models

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen if you feel it is still relevant. Thank you!