[Model] Refactoring of MiniCPM-V and add MiniCPM-o-2.6 support for vLLM

[HwwwwwwwH]

This PR aims to adapt and support all the features of MiniCPM-V and MiniCPM-o. It is designed to be compatible with various modalities (image, video, audio), different model versions (2.0, 2.5, 2.6, o), and diverse input types (raw, embeddings), while maintaining support for LORA, which might require significant effort.

Below is the roadmap for this PR:

• Refactor the input processor code of MiniCPM-V for MultiModalInputsV2 of vLLM.
  • Support for image and video inputs.
  • Support for image embeddings inputs.
  • Support for video embeddings inputs.
  • Previous supports for MiniCPM-o.
  • Verify LORA support.
• Adapt new features of MiniCPM-o.
  • Support for audio and audio embeddings inputs.
  • Support for image and audio interleave inputs.
  • Support for audio outputs (using hidden states)[Furture work].
  • Streaming multimodal inputs (may be complex; consider starting a new PR for this feature in the future)[Furture work].

This PR is still in development. Once I complete the support for audio, I will request to merge. I'll get this work done ASAP.

FIX #12162

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[ywang96]

Really appreciate your effort planned on this PR!

Support for audio outputs (using hidden states).
Streaming multimodal inputs (may be complex; consider starting a new PR for this feature in the future).

It would be great if you can share some design decisions for the these two items as an RFC (or two separate RFCs) first before we proceed with implementation. We (vLLM team) are also thinking about how we want to support multimodal output and streaming/realtime API on vLLM so it's probably the best time for us to discuss these items!

[HwwwwwwwH]

Really appreciate your effort planned on this PR!

Support for audio outputs (using hidden states).
Streaming multimodal inputs (may be complex; consider starting a new PR for this feature in the future).

It would be great if you can share some design decisions for the these two items as an RFC (or two separate RFCs) first before we proceed with implementation. We (vLLM team) are also thinking about how we want to support multimodal output and streaming/realtime API on vLLM so it's probably the best time for us to discuss these items!

Thank you for suggestion! I'll start these two RFCs tomorrow.

[HwwwwwwwH]

@DarkLight1337 I think I might need some help for verifying LoRA support. Should I do any changes for it?

[DarkLight1337]

@jeejeelee can help with this. Please keep in mind though that currently LoRA is only supported for the language part of multi-modal models.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @HwwwwwwwH.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[ywang96]

Overall LGTM and I have left a comment! I think this version is good to be merged!

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @HwwwwwwwH.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[HwwwwwwwH]

Finally get it merged, thanks for the help from @ywang96 @DarkLight1337 ! Then I 'll work on V1 support of MiniCPMV(O).

[jasstionzyf]

@HwwwwwwwH
Cannot use FlashAttention-2 backend for head size 72 for miniCPM-V 2.6
#12656
if can not use FlashAttention-2 , much slow when generate

[wangyuanxiong-hub]

The adaptation is timely, but the lack of support for streaming multimodal inputs currently results in slower vision inference speeds.

[wangyuanxiong-hub]

The adaptation is timely, but the lack of support for streaming multimodal inputs currently results in slower vision inference speeds.适应是及时的，但缺乏支持流的多式联运输入目前的结果在慢的愿景推断的速度。

Here is the transformer implementation https://github.com/thanhnienyeumeo/minicpm-o

[Jiltseb]

@HwwwwwwwH @ywang96

How is VLLM omni-mode support is implemented, multimodal support is a crucial part of this model.

With transformers below part implements it.

sys_msg = model.get_sys_prompt(mode='omni', language='en')
contents = get_video_chunk_content(video_path) #contains both audio and image 1sec contents TDM
msg = {"role":"user", "content": contents}

[HwwwwwwwH]

@HwwwwwwwH @ywang96

How is VLLM omni-mode support is implemented, multimodal support is a crucial part of this model.

With transformers below part implements it.

sys_msg = model.get_sys_prompt(mode='omni', language='en')
contents = get_video_chunk_content(video_path) #contains both audio and image 1sec contents TDM
msg = {"role":"user", "content": contents}

You need to split and concatenate video frames and audio chunks by your self and use prompt like:

[(<video>./</video>)(<audio>./</audio>)] * length

[Jiltseb]

@HwwwwwwwH Great, there will be a corresponding multimodal data containing multiple images and audios (1sec each). I got it working for a single image, audio pair via:

input_data = {
    "prompt": prompt,
    "multi_modal_data": contents[0]
}

where
contents = [{'image': Image obj, 'audio': audio_array},{'image': Image obj, 'audio': audio_array},{'image': Image obj, 'audio': audio_array},...]
How can I run a single generation with multiple image/audio pairs?

[wangyuanxiong-hub]

@HwwwwwwwH Great, there will be a corresponding multimodal data containing multiple images and audios (1sec each). I got it working for a single image, audio pair via:

input_data = {
    "prompt": prompt,
    "multi_modal_data": contents[0]
}

where contents = [{'image': Image obj, 'audio': audio_array},{'image': Image obj, 'audio': audio_array},{'image': Image obj, 'audio': audio_array},...] How can I run a single generation with multiple image/audio pairs?

same question