load model failed. 加载模型失败

[Wanger-SJTU]

Traceback (most recent call last):
  File "cli_demo.py", line 19, in <module>
    local_doc_qa.init_cfg(llm_model=LLM_MODEL,
  File "/dlcode/langchain-ChatGLM/chains/local_doc_qa.py", line 37, in init_cfg
    self.llm.load_model(model_name_or_path=llm_model_dict[llm_model],
  File "dlcode/langchain-ChatGLM/models/chatglm_llm.py", line 55, in load_model
    self.tokenizer = AutoTokenizer.from_pretrained(
  File "anaconda3/envs/chatGLM/lib/python3.8/site-packages/transformers/models/auto/tokenization_auto.py", line 679, in from_pretrained
    return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
  File "anaconda3/envs/chatGLM/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 1804, in from_pretrained
    return cls._from_pretrained(
  File "/anaconda3/envs/chatGLM/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 1959, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/.cache/huggingface/modules/transformers_modules/tokenization_chatglm.py", line 209, in __init__
    self.sp_tokenizer = SPTokenizer(vocab_file, num_image_tokens=num_image_tokens)
  File "/.cache/huggingface/modules/transformers_modules/tokenization_chatglm.py", line 61, in __init__
    self.text_tokenizer = TextTokenizer(vocab_file)
  File "/.cache/huggingface/modules/transformers_modules/tokenization_chatglm.py", line 22, in __init__
    self.sp.Load(model_path)
  File "/anaconda3/envs/chatGLM/lib/python3.8/site-packages/sentencepiece/__init__.py", line 906, in Load
    return self.LoadFromFile(model_file)
  File "/anaconda3/envs/chatGLM/lib/python3.8/site-packages/sentencepiece/__init__.py", line 310, in LoadFromFile
    return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
RuntimeError: Internal: src/sentencepiece_processor.cc(1101) [model_proto->ParseFromArray(serialized.data(), serialized.size())] 

[imClumsyPanda]

请问模型文件是否时最新版本？是否已按照 README 中介绍的环境要求，完成 pip install -r requirements.txt？

[Wanger-SJTU]

请问模型文件是否时最新版本？是否已按照 README 中介绍的环境要求，完成 pip install -r requirements.txt？

全部是最新的

[yanqiangmiffy]

你这个应该是transformers的环境没有安装好，你tranformers版本多少，另外stencepiece这个有没有安装好

[Wanger-SJTU]

你这个应该是transformers的环境没有安装好，你tranformers版本多少，另外stencepiece这个有没有安装好

$ pip list                       
Package                 Version
----------------------- ------------------
absl-py                 1.4.0
aiofiles                23.1.0
aiohttp                 3.8.4
aiosignal               1.3.1
altair                  4.2.2
antlr4-python3-runtime  4.9.3
anyio                   3.6.2
appdirs                 1.4.4
argilla                 1.6.0
async-timeout           4.0.2
attrs                   23.1.0
backoff                 2.2.1
beautifulsoup4          4.12.2
black                   21.4b2
brotlipy                0.7.0
cachetools              5.3.0
certifi                 2022.12.7
cffi                    1.15.1
charset-normalizer      2.0.4
click                   8.1.3
cloudpickle             2.2.1
coloredlogs             15.0.1
commonmark              0.9.1
contourpy               1.0.7
cpm-kernels             1.0.11
cryptography            39.0.1
cycler                  0.11.0
dataclasses-json        0.5.7
Deprecated              1.2.13
detectron2              0.6
effdet                  0.3.0
entrypoints             0.4
et-xmlfile              1.1.0
faiss-cpu               1.7.3
fastapi                 0.95.1
ffi-navigator           0.5.3
ffmpy                   0.3.0
filelock                3.11.0
flatbuffers             23.3.3
flit_core               3.8.0
fonttools               4.39.3
frozenlist              1.3.3
fsspec                  2023.4.0
future                  0.18.3
fvcore                  0.1.5.post20221221
google-auth             2.17.3
google-auth-oauthlib    0.4.6
gptcache                0.1.14
gradio                  3.27.0
gradio_client           0.1.3
greenlet                2.0.2
grpcio                  1.53.0
h11                     0.14.0
httpcore                0.16.3
httpx                   0.23.3
huggingface-hub         0.13.4
humanfriendly           10.0
hydra-core              1.3.2
icetk                   0.0.7
idna                    3.4
importlib-metadata      6.4.1
importlib-resources     5.12.0
iopath                  0.1.9
Jinja2                  3.1.2
joblib                  1.2.0
jsonschema              4.17.3
kiwisolver              1.4.4
langchain               0.0.142
layoutparser            0.3.4
linkify-it-py           2.0.0
lxml                    4.9.2
Markdown                3.4.3
markdown-it-py          2.2.0
MarkupSafe              2.1.2
marshmallow             3.19.0
marshmallow-enum        1.5.1
matplotlib              3.7.1
mdit-py-plugins         0.3.3
mdurl                   0.1.2
mkl-fft                 1.3.1
mkl-random              1.2.2
mkl-service             2.4.0
monotonic               1.6
mpmath                  1.3.0
msg-parser              1.2.0
multidict               6.0.4
mypy-extensions         1.0.0
nltk                    3.8.1
numexpr                 2.8.4
numpy                   1.23.5
oauthlib                3.2.2
olefile                 0.46
omegaconf               2.3.0
onnxruntime             1.14.1
openai                  0.27.4
openapi-schema-pydantic 1.2.4
opencv-python           4.6.0.66
openpyxl                3.1.2
orjson                  3.8.10
packaging               23.1
pandas                  1.5.3
pathspec                0.11.1
pdf2image               1.16.3
pdfminer.six            20221105
pdfplumber              0.9.0
Pillow                  9.4.0
pip                     23.0.1
pkgutil_resolve_name    1.3.10
portalocker             2.7.0
protobuf                3.18.3
pyasn1                  0.4.8
pyasn1-modules          0.2.8
pycocotools             2.0.6
pycparser               2.21
pydantic                1.10.7
pydot                   1.4.2
pydub                   0.25.1
Pygments                2.15.0
pyOpenSSL               23.0.0
pypandoc                1.11
pyparsing               3.0.9
pyrsistent              0.19.3
PySocks                 1.7.1
pytesseract             0.3.10
python-dateutil         2.8.2
python-docx             0.8.11
python-magic            0.4.27
python-multipart        0.0.6
python-pptx             0.6.21
pytz                    2023.3
PyYAML                  6.0
regex                   2023.3.23
requests                2.28.1
requests-oauthlib       1.3.1
rfc3986                 1.5.0
rich                    13.0.1
rsa                     4.9
scikit-learn            1.2.2
scipy                   1.10.1
semantic-version        2.10.0
sentence-transformers   2.2.2
sentencepiece           0.1.98
setuptools              66.0.0
six                     1.16.0
sniffio                 1.3.0
soupsieve               2.4.1
SQLAlchemy              1.4.47
starlette               0.26.1
sympy                   1.11.1
tabulate                0.9.0
tenacity                8.2.2
tensorboard             2.11.2
tensorboard-data-server 0.6.1
tensorboard-plugin-wit  1.8.1
termcolor               2.2.0
threadpoolctl           3.1.0
timm                    0.6.13
tokenizers              0.13.3
toml                    0.10.2
toolz                   0.12.0
torch                   1.12.1
torchaudio              0.12.1
torchvision             0.13.1
tqdm                    4.65.0
transformers            4.27.1
typing_extensions       4.4.0
typing-inspect          0.8.0
uc-micro-py             1.0.1
unstructured            0.5.12
unstructured-inference  0.3.2
urllib3                 1.26.15
uvicorn                 0.21.1
Wand                    0.6.11
websockets              11.0.2
Werkzeug                2.2.3
wheel                   0.38.4
wrapt                   1.14.1
XlsxWriter              3.1.0
yacs                    0.1.8
yarl                    1.8.2
zipp                    3.15.0

@yanqiangmiffy

[yanqiangmiffy]

huggingface/transformers#20011

你升级下transformers的版本到最新的28.1，然后你chatglm-6b的模型文件很旧了吧，可能词汇表变了，需要注意下，官方更新了好几次版本了

[Wanger-SJTU]

chatGLM 仓文件下载有问题，已解决。

[CELTIANqzq]

请问一下chatGLM 仓文件下载有问题是指哪部分下载有问题，我遇到了和你一样的情况

[Wanger-SJTU]

请问一下chatGLM 仓文件下载有问题是指哪部分下载有问题，我遇到了和你一样的情况

我是ice_text.model模型没有下载下来。开启lfs，再下载hugging face的代码仓,你先看下glm的下载下来ice_text.model这个模型了没有

[zdgithub]

我也遇到了相同的问题，下载的chatglm2-6b，这个模型原本就没有ice_text.model这个文件啊