Add support for GLM-Edge and GLM-Edge-V series models

[piDack]

This pull request support for the GLM-Edge-Chat 1.5B & 4B and GLM-Edge-V 2B & 5B series of models within the llama.cpp.

Note: The current model pretrain -> gguf only supports using the transformers version 4.47.0.dev0.

• I have read the contributing guidelines
• Self-reported review complexity:
  • Low
  • Medium
  • High

[arch-btw]

Works great.

./llama-llava-cli -m ggml-model-Q4_K_M.gguf --mmproj mmproj-model-f16.gguf --temp 0.1 --image bee.jpg -p "<|system|>\n You are a helpful AI assistant. <image><|user|>\n What is in the image? <|assistant|>\n"

[piDack]

Is there anyone available to review the code?

[arch-btw]

@piDack As this would be adding support for GlmForCausalLM for these vision models, I'm curious to know if we could create a more modular or generic implementation that could also be used for the other GlmForCausalLM model(s)?

I'm asking because glm-4-9b-chat-hf is currently broken with the new transformers-only implementation:

python convert_hf_to_gguf.py /home/Models/glm-4-9b-chat-hf --outtype f32
INFO:hf-to-gguf:Loading model: glm-4-9b-chat-hf
ERROR:hf-to-gguf:Model GlmForCausalLM is not supported

The version with the custom python files still works but if we're moving away from that (related discussion), it might be best to support GlmForCausalLM in general.

Are there any parts of this PR that could be refactored or generalized for broader applicability so that we can support both and maybe upcoming models? Thank you.

[piDack]

@piDack As this would be adding support for GlmForCausalLM for these vision models, I'm curious to know if we could create a more modular or generic implementation that could also be used for the other GlmForCausalLM model(s)?

I'm asking because glm-4-9b-chat-hf is currently broken with the new transformers-only implementation:

python convert_hf_to_gguf.py /home/Models/glm-4-9b-chat-hf --outtype f32
INFO:hf-to-gguf:Loading model: glm-4-9b-chat-hf
ERROR:hf-to-gguf:Model GlmForCausalLM is not supported

The version with the custom python files still works but if we're moving away from that (related discussion), it might be best to support GlmForCausalLM in general.

Are there any parts of this PR that could be refactored or generalized for broader applicability so that we can support both and maybe upcoming models? Thank you.

I will tried to do it

[piDack]

@piDack As this would be adding support for GlmForCausalLM for these vision models, I'm curious to know if we could create a more modular or generic implementation that could also be used for the other GlmForCausalLM model(s)?

I'm asking because glm-4-9b-chat-hf is currently broken with the new transformers-only implementation:

python convert_hf_to_gguf.py /home/Models/glm-4-9b-chat-hf --outtype f32
INFO:hf-to-gguf:Loading model: glm-4-9b-chat-hf
ERROR:hf-to-gguf:Model GlmForCausalLM is not supported

The version with the custom python files still works but if we're moving away from that (related discussion), it might be best to support GlmForCausalLM in general.

Are there any parts of this PR that could be refactored or generalized for broader applicability so that we can support both and maybe upcoming models? Thank you.

done

[arch-btw]

Perfect! Thank you @piDack
@ngxson could you please take a look?

[ngxson]

many places still having inconsistent style

[ngxson]

@ggerganov Could you have a look quickly on llama.cpp? I gave my approval for the rest of changes.

[piDack]

I believe it's ready to be merged into the master branch.