[BUG] RuntimeError: Internal: could not parse ModelProto from chatglm3-6b/tokenizer.model #3103

ZhichaoOuyang · 2024-02-28T08:25:19Z

运行 python startup.py -a

报错：
==============================Langchain-Chatchat Configuration==============================
操作系统：Linux-4.18.0-2.4.3.kwai.x86_64-x86_64-with-glibc2.27.
python版本：3.11.5 (main, Sep 11 2023, 13:54:46) [GCC 11.2.0]
项目版本：v0.2.10
langchain版本：0.0.354. fastchat版本：0.2.35

当前使用的分词器：ChineseRecursiveTextSplitter
当前启动的LLM模型：['chatglm3-6b', 'zhipu-api', 'openai-api'] @ cpu
{'device': 'cuda',
'host': '0.0.0.0',
'infer_turbo': False,
'model_path': 'chatglm3-6b',
'model_path_exists': True,
'port': 20002}
{'api_key': '',
'device': 'auto',
'host': '0.0.0.0',
'infer_turbo': False,
'online_api': True,
'port': 21001,
'provider': 'ChatGLMWorker',
'version': 'glm-4',
'worker_class': <class 'server.model_workers.zhipu.ChatGLMWorker'>}
{'api_base_url': 'https://api.openai.com/v1',
'api_key': '',
'device': 'auto',
'host': '0.0.0.0',
'infer_turbo': False,
'model_name': 'gpt-4',
'online_api': True,
'openai_proxy': '',
'port': 20002}
当前Embbedings模型： bge-large-zh-v1.5 @ cpu
==============================Langchain-Chatchat Configuration==============================

2024-02-28 08:16:53,771 - startup.py[line:655] - INFO: 正在启动服务：
2024-02-28 08:16:53,771 - startup.py[line:656] - INFO: 如需查看 llm_api 日志，请前往 /home/ouyangzhichao/Langchain-Chatchat/logs
/home/ouyangzhichao/anaconda3/lib/python3.11/site-packages/langchain_core/_api/deprecation.py:117: LangChainDeprecationWarning: 模型启动功能将于 Langchain-Chatchat 0.3.x重写,支持更多模式和加速启动，0.2.x中相关功能将废弃
warn_deprecated(
2024-02-28 08:17:01 | INFO | model_worker | Register to controller
2024-02-28 08:17:01 | ERROR | stderr | INFO: Started server process [3160451]
2024-02-28 08:17:01 | ERROR | stderr | INFO: Waiting for application startup.
2024-02-28 08:17:01 | ERROR | stderr | INFO: Application startup complete.
2024-02-28 08:17:01 | ERROR | stderr | INFO: Uvicorn running on http://0.0.0.0:20000 (Press CTRL+C to quit)
2024-02-28 08:17:01 | INFO | model_worker | Loading the model ['chatglm3-6b'] on worker 05c9ab76 ...
2024-02-28 08:17:02 | ERROR | stderr | Process model_worker - chatglm3-6b:
2024-02-28 08:17:02 | ERROR | stderr | Traceback (most recent call last):
2024-02-28 08:17:02 | ERROR | stderr | File "/home/ouyangzhichao/anaconda3/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
2024-02-28 08:17:02 | ERROR | stderr | self.run()
2024-02-28 08:17:02 | ERROR | stderr | File "/home/ouyangzhichao/anaconda3/lib/python3.11/multiprocessing/process.py", line 108, in run
2024-02-28 08:17:02 | ERROR | stderr | self._target(*self._args, **self._kwargs)
2024-02-28 08:17:02 | ERROR | stderr | File "/home/ouyangzhichao/Langchain-Chatchat/startup.py", line 389, in run_model_worker
2024-02-28 08:17:02 | ERROR | stderr | app = create_model_worker_app(log_level=log_level, **kwargs)
2024-02-28 08:17:02 | ERROR | stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-02-28 08:17:02 | ERROR | stderr | File "/home/ouyangzhichao/Langchain-Chatchat/startup.py", line 217, in create_model_worker_app
2024-02-28 08:17:02 | ERROR | stderr | worker = ModelWorker(
2024-02-28 08:17:02 | ERROR | stderr | ^^^^^^^^^^^^
2024-02-28 08:17:02 | ERROR | stderr | File "/home/ouyangzhichao/anaconda3/lib/python3.11/site-packages/fastchat/serve/model_worker.py", line 77, in init
2024-02-28 08:17:02 | ERROR | stderr | self.model, self.tokenizer = load_model(
2024-02-28 08:17:02 | ERROR | stderr | ^^^^^^^^^^^
2024-02-28 08:17:02 | ERROR | stderr | File "/home/ouyangzhichao/anaconda3/lib/python3.11/site-packages/fastchat/model/model_adapter.py", line 348, in load_model
2024-02-28 08:17:02 | ERROR | stderr | model, tokenizer = adapter.load_model(model_path, kwargs)
2024-02-28 08:17:02 | ERROR | stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-02-28 08:17:02 | ERROR | stderr | File "/home/ouyangzhichao/anaconda3/lib/python3.11/site-packages/fastchat/model/model_adapter.py", line 816, in load_model
2024-02-28 08:17:02 | ERROR | stderr | tokenizer = AutoTokenizer.from_pretrained(
2024-02-28 08:17:02 | ERROR | stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-02-28 08:17:02 | ERROR | stderr | File "/home/ouyangzhichao/anaconda3/lib/python3.11/site-packages/transformers/models/auto/tokenization_auto.py", line 801, in from_pretrained
2024-02-28 08:17:02 | ERROR | stderr | return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
2024-02-28 08:17:02 | ERROR | stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-02-28 08:17:02 | ERROR | stderr | File "/home/ouyangzhichao/anaconda3/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 2029, in from_pretrained
2024-02-28 08:17:02 | ERROR | stderr | return cls._from_pretrained(
2024-02-28 08:17:02 | ERROR | stderr | ^^^^^^^^^^^^^^^^^^^^^
2024-02-28 08:17:02 | ERROR | stderr | File "/home/ouyangzhichao/anaconda3/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 2261, in _from_pretrained
2024-02-28 08:17:02 | ERROR | stderr | tokenizer = cls(*init_inputs, **init_kwargs)
2024-02-28 08:17:02 | ERROR | stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-02-28 08:17:02 | ERROR | stderr | File "/home/ouyangzhichao/.cache/huggingface/modules/transformers_modules/THUDM/chatglm3-6b/9addbe01105ca1939dd60a0e5866a1812be9daea/tokenization_chatglm.py", line 109, in init
2024-02-28 08:17:02 | ERROR | stderr | self.tokenizer = SPTokenizer(vocab_file)
2024-02-28 08:17:02 | ERROR | stderr | ^^^^^^^^^^^^^^^^^^^^^^^
2024-02-28 08:17:02 | ERROR | stderr | File "/home/ouyangzhichao/.cache/huggingface/modules/transformers_modules/THUDM/chatglm3-6b/9addbe01105ca1939dd60a0e5866a1812be9daea/tokenization_chatglm.py", line 18, in init
2024-02-28 08:17:02 | ERROR | stderr | self.sp_model = SentencePieceProcessor(model_file=model_path)
2024-02-28 08:17:02 | ERROR | stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-02-28 08:17:02 | ERROR | stderr | File "/home/ouyangzhichao/anaconda3/lib/python3.11/site-packages/sentencepiece/init.py", line 468, in Init
2024-02-28 08:17:02 | ERROR | stderr | self.Load(model_file=model_file, model_proto=model_proto)
2024-02-28 08:17:02 | ERROR | stderr | File "/home/ouyangzhichao/anaconda3/lib/python3.11/site-packages/sentencepiece/init.py", line 961, in Load
2024-02-28 08:17:02 | ERROR | stderr | return self.LoadFromFile(model_file)
2024-02-28 08:17:02 | ERROR | stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-02-28 08:17:02 | ERROR | stderr | File "/home/ouyangzhichao/anaconda3/lib/python3.11/site-packages/sentencepiece/init.py", line 316, in LoadFromFile
2024-02-28 08:17:02 | ERROR | stderr | return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
2024-02-28 08:17:02 | ERROR | stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-02-28 08:17:02 | ERROR | stderr | RuntimeError: Internal: could not parse ModelProto from chatglm3-6b/tokenizer.model

前置步骤都按照要求搞了，求大佬帮忙看看

dosubot · 2024-02-28T08:25:37Z

🤖

The error message "RuntimeError: Internal: could not parse ModelProto from chatglm3-6b/tokenizer.model" suggests that there is an issue with loading the tokenizer model for the chatglm3-6b model. This could be due to several reasons, such as the model file being corrupted, missing, or incompatible with the current setup.

To resolve this issue, follow these steps:

Verify Model Path: Ensure that the chatglm3-6b model and its tokenizer are correctly placed in the directory specified by MODEL_ROOT_PATH in your model_config.py.example file. If MODEL_ROOT_PATH is empty, the model should be in a directory that matches its key in the MODEL_PATH dictionary, relative to the script's running directory.
Check Model Integrity: Make sure that the chatglm3-6b model files, especially the tokenizer model file, are not corrupted. You might want to re-download or obtain a fresh copy of the model and its tokenizer.
Compatibility Check: Ensure that the model and tokenizer are compatible with the libraries and Python version you are using. Since you are using Python version 3.11.5, verify that all dependencies (like TensorFlow, PyTorch, or any other relevant library) are up-to-date and compatible with Python 3.11.5.
Update Dependencies: Sometimes, updating your dependencies can resolve compatibility issues. Make sure that your environment has the latest versions of necessary libraries (e.g., Hugging Face Transformers, if used).
Model Configuration: Double-check the model_config.py.example file to ensure that the path to the chatglm3-6b model and its tokenizer is correctly specified. If you have modified the default paths or names, ensure that these modifications are accurately reflected in the configuration.

If after following these steps the issue persists, it might be helpful to look for any specific documentation or community posts related to loading the chatglm3-6b model in the context of your project or the libraries you are using.

To load the chatglm3-6b model in the Langchain-Chatchat project, follow these steps:

Ensure you have the model file for chatglm3-6b. According to the provided model_config.py.example file, the model identifier for chatglm3-6b is "THUDM/chatglm3-6b". This suggests that the model is hosted on Hugging Face. You should download the model from Hugging Face or ensure it's accessible in your environment.
Set the MODEL_ROOT_PATH in the model_config.py.example file to the directory where you store all your models. If you place the chatglm3-6b model directly under this directory or within a subdirectory named after the model, the program will automatically detect and load it without needing to modify the MODEL_PATH dictionary for llm_model.
In the LLM_MODELS list, chatglm3-6b is already included and is set as the first model, making it the default model for the API and WEBUI. If it wasn't, you would add "chatglm3-6b" to the list to include it for loading at project startup.
Ensure the LLM_DEVICE is set appropriately for your hardware. It's set to "auto" by default, which automatically detects the hardware but may issue a warning. You can manually set it to "cuda", "mps", "cpu", or "xpu" depending on your system's capabilities and requirements.
Rename model_config.py.example to model_config.py after making the necessary adjustments.

Here's a summary of the key configurations from your context:

MODEL_ROOT_PATH = ""  # Set this to your models' directory path

LLM_MODELS = ["chatglm3-6b", "zhipu-api", "openai-api"]  # chatglm3-6b is already set as the default model

LLM_DEVICE = "auto"  # Adjust if necessary for your hardware

MODEL_PATH = {
    "llm_model": {
        "chatglm3-6b": "THUDM/chatglm3-6b",
        # other models...
    },
    # other model configurations...
}

Ensure you have the chatglm3-6b model stored correctly in the directory specified by MODEL_ROOT_PATH or accessible online through Hugging Face with the identifier "THUDM/chatglm3-6b".

For more information, you can refer to the model_config.py.example file in the Langchain-Chatchat repository.

Sources

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

zRzRzRzRzRzRzR · 2024-02-28T12:02:05Z

你模型是在hf下载的么，是不是模型没全

ZhichaoOuyang · 2024-02-29T02:09:25Z

你模型是在hf下载的么，是不是模型没全

对，是不全的问题，重新下解决了

liyiyiya · 2024-04-23T04:10:13Z

运行 python startup.py -a

报错： ==============================Langchain-Chatchat Configuration============================== 操作系统：Linux-4.18.0-2.4.3.kwai.x86_64-x86_64-with-glibc2.27. python版本：3.11.5 （main， Sep 11 2023， 13：54：46） [GCC 11.2.0] 项目版本：v0.2.10 langchain版本：0.0.354.fastchat版本：0.2.35

当前使用的分词器：ChineseRecursiveTextSplitter 当前启动的LLM模型：['chatglm3-6b'， 'zhipu-api'， 'openai-api'] @ cpu {'device'： 'cuda'， 'host'： '0.0.0.0'， 'infer_turbo'： False， 'model_path'： 'chatglm3-6b'， 'model_path_exists'： True， 'port'： 20002} {'api_key'： ''， 'device'： 'auto'， 'host'： '0.0.0.0'， 'infer_turbo'： False， 'online_api'： True， 'port'： 21001， 'provider'： 'ChatGLMWorker'， 'version'： 'glm-4'， 'worker_class'： <class 'server.model_workers.zhipu.ChatGLMWorker'>} {'api_base_url'： 'https://api.openai.com/v1'， 'api_key'： ''， 'device'： 'auto'， 'host'： '0.0.0.0'， 'infer_turbo'： False， 'model_name'： 'gpt-4'， 'online_api'： True， 'openai_proxy'： ''， 'port'： 20002} 当前Embbedings模型： bge-large-zh-v1.5 @ cpu ==============================Langchain-Chatchat Configuration==============================

2024-02-28 08：16：53,771 - startup.py[line：655] - INFO：正在启动服务： 2024-02-28 08：16：53,771 - startup.py[line：656] - INFO：如需查看 llm_api 日志，请前往 /home/ouyangzhichao/langchain-chatchat/logs /home/ouyangzhichao/anaconda3/lib/python3.11/site-packages/langchain_core/_api/deprecation.py：117： LangChainDeprecationWarning：模型启动功能将于 Langchain-Chatchat 0.3.x重写，支持更多模式和加速启动，0.2.x中相关功能将废弃 warn_deprecated（ 2024-02-28 08：17：01 |资讯 |model_worker |注册控制器 2024-02-28 08：17：01 |错误 |斯特德尔 |信息：服务器进程已启动 [3160451] 2024-02-28 08：17：01 |错误 |斯特德尔 |信息：正在等待应用程序启动。 2024-02-28 08：17：01 |错误 |斯特德尔 |信息：应用程序启动完成。 2024-02-28 08：17：01 |错误 |斯特德尔 |信息：Uvicorn 在 http://0.0.0.0:20000 上运行（按 CTRL+C 退出） 2024-02-28 08：17：01 |资讯 |model_worker |在工作器 05c9ab76 上加载模型 ['chatglm3-6b'] ... 2024-02-28 08：17：02 |错误 |斯特德尔 |流程model_worker - chatglm3-6b： 2024-02-28 08：17：02 |错误 |斯特德尔 |回溯（最近一次调用最后）：2024-02-28 08：17：02 |错误 |斯特德尔 |文件“/home/ouyangzhichao/anaconda3/lib/python3.11/multiprocessing/process.py”，第 314 行，在 _bootstrap 2024-02-28 08：17：02 |错误 |斯特德尔 |自我运行（） 2024-02-28 08：17：02 |错误 |斯特德尔 |文件“/home/ouyangzhichao/anaconda3/lib/python3.11/multiprocessing/process.py”，第 108 行，正在运行 2024-02-28 08：17：02 |错误 |斯特德尔 |self._target（*self._args， **self._kwargs） 2024-02-28 08：17：02 |错误 |斯特德尔 |文件“/home/ouyangzhichao/langchain-chatchat/startup.py”，第 389 行，在 run_model_worker 2024-02-28 08：17：02 |错误 |斯特德尔 |app = create_model_worker_app（log_level=log_level， **kwargs） 2024-02-28 08：17：02 |错误 |斯特德尔 |^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-02-28 08：17：02 |错误 |斯特德尔 |文件“/home/ouyangzhichao/langchain-chatchat/startup.py”，第 217 行，在 create_model_worker_app 2024-02-28 08：17：02 |错误 |斯特德尔 |worker = 模型工人（ 2024-02-28 08：17：02 |错误 |斯特德尔 |^^^^^^^^^^^^ 2024-02-28 08：17：02 |错误 |斯特德尔 |文件“/home/ouyangzhichao/anaconda3/lib/python3.11/site-packages/fastchat/serve/model_worker.py”，第 77 行，init 2024-02-28 08：17：02 |错误 |斯特德尔 |self.model， self.tokenizer = load_model（ 2024-02-28 08：17：02 |错误 |斯特德尔 |^^^^^^^^^^^ 2024-02-28 08：17：02 |错误 |斯特德尔 |文件“/home/ouyangzhichao/anaconda3/lib/python3.11/site-packages/fastchat/model/model_adapter.py”，第 348 行，load_model 2024-02-28 08：17：02 |错误 |斯特德尔 |模型，分词器 = adapter.load_model（model_path， kwargs） 2024-02-28 08：17：02 |错误 |斯特德尔 |^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-02-28 08：17：02 |错误 |斯特德尔 |文件“/home/ouyangzhichao/anaconda3/lib/python3.11/site-packages/fastchat/model/model_adapter.py”，第 816 行，在 load_model 2024-02-28 08：17：02 |错误 |斯特德尔 |分词器 = AutoTokenizer.from_pretrained（ 2024-02-28 08：17：02 |错误 |斯特德尔 |^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-02-28 08：17：02 |错误 |斯特德尔 |文件 “/home/ouyangzhichao/anaconda3/lib/python3.11/site-packages/transformers/models/auto/tokenization_auto.py“，801行，在from_pretrained 2024-02-28 08：17：02 |错误 |斯特德尔 |返回 tokenizer_class.from_pretrained（pretrained_model_name_or_path， *inputs， **kwargs） 2024-02-28 08：17：02 |错误 |斯特德尔 |^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-02-28 08：17：02 |错误 |斯特德尔 |文件“/home/ouyangzhichao/anaconda3/lib/python3.11/site-packages/transformers/tokenization_utils_base.py”，第 2029 行，在 from_pretrained 2024-02-28 08：17：02 |错误 |斯特德尔 |返回cls._from_pretrained（ 2024-02-28 08：17：02 |错误 |斯特德尔 |^^^^^^^^^^^^^^^^^^^^^ 2024-02-28 08：17：02 |错误 |斯特德尔 |文件“/home/ouyangzhichao/anaconda3/lib/python3.11/site-packages/transformers/tokenization_utils_base.py”，第 2261 行，_from_pretrained 2024-02-28 08：17：02 |错误 |斯特德尔 |分词器 = cls（*init_inputs， **init_kwargs） 2024-02-28 08：17：02 |错误 |斯特德尔 |^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-02-28 08：17：02 |错误 |斯特德尔 |文件“/home/ouyangzhichao/.cache/huggingface/modules/transformers_modules/THUDM/chatglm3-6b/9addbe01105ca1939dd60a0e5866a1812be9daea/tokenization_chatglm.py”，第 109 行，init 2024-02-28 08：17：02 |错误 |斯特德尔 |self.tokenizer = SPTokenizer（vocab_file） 2024-02-28 08：17：02 |错误 |斯特德尔 |^^^^^^^^^^^^^^^^^^^^^^^ 2024-02-28 08：17：02 |错误 |斯特德尔 |文件“/home/ouyangzhichao/.cache/huggingface/modules/transformers_modules/THUDM/chatglm3-6b/9addbe01105ca1939dd60a0e5866a1812be9daea/tokenization_chatglm.py”，第 18 行，init 2024-02-28 08：17：02 |错误 |斯特德尔 |self.sp_model = SentencePieceProcessor（model_file=model_path） 2024-02-28 08：17：02 |错误 |斯特德尔 |^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-02-28 08：17：02 |错误 |斯特德尔 |文件“/home/ouyangzhichao/anaconda3/lib/python3.11/site-packages/sentencepiece/init.py”，第 468 行，在 Init 中 2024-02-28 08：17：02 |错误 |斯特德尔 |自我。负载（model_file=model_file， model_proto=model_proto） 2024-02-28 08：17：02 |错误 |斯特德尔 |文件“/home/ouyangzhichao/anaconda3/lib/python3.11/site-packages/sentencepiece/init.py”，第 961 行，加载 2024-02-28 08：17：02 |错误 |斯特德尔 |返回自我。LoadFromFile（model_file） 2024-02-28 08：17：02 |错误 |斯特德尔 |^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-02-28 08：17：02 |错误 |斯特德尔 |文件“/home/ouyangzhichao/anaconda3/lib/python3.11/site-packages/sentencepiece/init.py”，第 316 行，在 LoadFromFile 2024-02-28 08：17：02 |错误 |斯特德尔 |返回_sentencepiece。SentencePieceProcessor_LoadFromFile（self， arg） 2024-02-28 08：17：02 |错误 |斯特德尔 |^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-02-28 08：17：02 |错误 |斯特德尔 |RuntimeError：内部：无法从 chatglm3-6b/tokenizer.model 解析 ModelProto

前置步骤都按照要求搞了，求大佬帮忙看看

兄弟，你是怎么解决的呀，我也出现这个问题了

liyiyiya · 2024-04-23T04:11:01Z

你模型是在hf下载的么，是不是模型没全

对，是不全的问题，重新下解决了

兄弟怎么重新下呀？跟着文档重新来一遍吗？还需要初始化吗？

liyiyiya · 2024-04-23T04:14:33Z

我的是这样，这是全的吗。我看不缺呀

swordqiqi · 2024-04-25T03:03:57Z

哥们，我遇到了同样的问题，你试试重新上传一下tokenizer.model能不能解决。

biu0924 · 2024-07-13T11:32:45Z

同问，怎么个不全法啊，我重新下了很多次了

Stevenzwzhai · 2024-07-25T07:07:58Z

模型要下载，一般来说直接git clone很快结束是不会下载完10多G的模型的

lixuancn · 2024-08-13T06:15:43Z

本质是没有git lfs，安装一下就好了。https://git-lfs.com/

zejun-chen · 2025-01-09T03:33:56Z

我也遇到了这个问题，llama3预训练，比较过tokenizer json文件的sha256sum，完全一致，说明数据完整性没问题，不需要重新下载的
现在仍然不能被parse，不知道要怎么做？
出错的log如下：

    self.Load(model_file=model_file, model_proto=model_proto)
  File "/home/sdp/miniforge3/envs/zejun_llama_train/lib/python3.10/site-packages/sentencepiece/__init__.py", line 963, in Load
    return self.LoadFromFile(model_file)
  File "/home/sdp/miniforge3/envs/zejun_llama_train/lib/python3.10/site-packages/sentencepiece/__init__.py", line 316, in LoadFromFile
    return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
**_RuntimeError: Internal: could not parse ModelProto from /mnt/frameworks.ai.pytorch.gpu-models.lfs-data/datasets/red_pajama_1T_llama3tok_sample/llama3_tokenizer/tokenizer.json_**

ZhichaoOuyang added the bug Something isn't working label Feb 28, 2024

zRzRzRzRzRzRzR self-assigned this Feb 28, 2024

ZhichaoOuyang closed this as completed Feb 29, 2024

dosubot bot mentioned this issue Mar 14, 2024

[BUG] dev3版本问英文问题时，会报错，问中文问题不会报错 #3326

Closed

dosubot bot mentioned this issue Mar 23, 2024

[BUG] 使用LLM模型问答报错（ --all-api 模式启动） #3491

Closed

This was referenced Apr 1, 2024

[BUG] 无法加载本地的chatglm3-6b，但是chatglm2-6b可以 #3583

Closed

Agent 不会调用工具 #3601

Closed

dosubot bot mentioned this issue Apr 10, 2024

运行python startup.py -a时报错MetadataIncompleteBuffer #3697

Closed

hiyouga mentioned this issue Apr 11, 2024

RuntimeError: Internal: could not parse ModelProto from /root/autodl-tmp/chatglm3-6b/tokenizer.model hiyouga/LLaMA-Factory#3229

Closed

1 task

dosubot bot mentioned this issue Apr 12, 2024

运行 python startup.py -a 加载chatglm3-6b模型到71%卡住不动 #3734

Closed

Cecilia-zwq mentioned this issue Apr 18, 2024

RuntimeError: Internal: could not parse ModelProto from /home/nlp/miniconda3/lib/python3.9/site-packages/inltk/models/hi/tokenizer.model goru001/inltk#99

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] RuntimeError: Internal: could not parse ModelProto from chatglm3-6b/tokenizer.model #3103

[BUG] RuntimeError: Internal: could not parse ModelProto from chatglm3-6b/tokenizer.model #3103

ZhichaoOuyang commented Feb 28, 2024

dosubot bot commented Feb 28, 2024 •

edited

Loading

About Dosu

zRzRzRzRzRzRzR commented Feb 28, 2024

ZhichaoOuyang commented Feb 29, 2024

liyiyiya commented Apr 23, 2024

liyiyiya commented Apr 23, 2024

liyiyiya commented Apr 23, 2024

swordqiqi commented Apr 25, 2024

biu0924 commented Jul 13, 2024

Stevenzwzhai commented Jul 25, 2024

lixuancn commented Aug 13, 2024

zejun-chen commented Jan 9, 2025

[BUG] RuntimeError: Internal: could not parse ModelProto from chatglm3-6b/tokenizer.model #3103

[BUG] RuntimeError: Internal: could not parse ModelProto from chatglm3-6b/tokenizer.model #3103

Comments

ZhichaoOuyang commented Feb 28, 2024

dosubot bot commented Feb 28, 2024 • edited Loading

Sources

About Dosu

zRzRzRzRzRzRzR commented Feb 28, 2024

ZhichaoOuyang commented Feb 29, 2024

liyiyiya commented Apr 23, 2024

liyiyiya commented Apr 23, 2024

liyiyiya commented Apr 23, 2024

swordqiqi commented Apr 25, 2024

biu0924 commented Jul 13, 2024

Stevenzwzhai commented Jul 25, 2024

lixuancn commented Aug 13, 2024

zejun-chen commented Jan 9, 2025

dosubot bot commented Feb 28, 2024 •

edited

Loading