Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[VLM] Merged multi-modal processor for InternVL-based models #12553

Merged
merged 26 commits into from
Feb 4, 2025
Merged

[VLM] Merged multi-modal processor for InternVL-based models #12553

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
efb0205
[VLM] Merged multi-modal processor for InternVL
DarkLight1337 Jan 29, 2025
883f9ed
Update tests
DarkLight1337 Jan 29, 2025
ff77a90
Fix multi-image input and update dummy input
DarkLight1337 Jan 29, 2025
5604ec2
Fix incorrect number of tokens
DarkLight1337 Jan 29, 2025
5132b51
Fix max tokens
DarkLight1337 Jan 29, 2025
a47f92a
Update processor tests
DarkLight1337 Jan 30, 2025
3814c90
Update H2OVL
DarkLight1337 Jan 30, 2025
ac6e6e2
Fix NVLM and H2O-VL
DarkLight1337 Jan 30, 2025
9b177b5
Update and try to fix tests
DarkLight1337 Jan 30, 2025
0a45104
Maintain correct alphabetical order
DarkLight1337 Jan 30, 2025
7466ce8
Remove outdated comment
DarkLight1337 Jan 30, 2025
bd67b1a
fix nvlm-d max_image_tokens
Isotr0py Jan 31, 2025
e287265
calculate number of non-image token in NVLM-D get_max_image_tokens
Isotr0py Jan 31, 2025
e2cc798
Update calculations
DarkLight1337 Feb 1, 2025
db19b6a
Fix
DarkLight1337 Feb 1, 2025
2ab4826
Pass `mm_counts` to `get_mm_max_tokens_per_item`
DarkLight1337 Feb 1, 2025
d764760
Fix
DarkLight1337 Feb 1, 2025
15d55a1
Fix
DarkLight1337 Feb 1, 2025
c17acb7
Clean up dtype
DarkLight1337 Feb 2, 2025
a8d3ed7
Fix H2O-VL
DarkLight1337 Feb 3, 2025
a636b3d
Avoid multiple warnings
DarkLight1337 Feb 3, 2025
5178a2b
Use the hardcoded value
DarkLight1337 Feb 3, 2025
701e970
Merge branch 'main' into internvl-v1
DarkLight1337 Feb 3, 2025
9e387cf
Fix missing license headers
DarkLight1337 Feb 3, 2025
83784c5
Merge branch 'main' into internvl-v1
DarkLight1337 Feb 4, 2025
bee5051
Update doc
DarkLight1337 Feb 4, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/source/models/supported_models.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -723,7 +723,7 @@ See [this page](#generative-models) for more information on how to use generativ
* ✅︎
- * `NVLM_D_Model`
* NVLM-D 1.0
* T + I<sup>E+</sup>
* T + I<sup>+</sup>
Copy link
Member Author

@DarkLight1337 DarkLight1337 Jan 29, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Embedding inputs is not supported because we need image size information to calculate the number of patches for the multi-modal processor.

* `nvidia/NVLM-D-72B`, etc.
*
* ✅︎
Expand Down
1 change: 0 additions & 1 deletion tests/models/decoder_only/vision_language/test_models.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -281,7 +281,6 @@
dtype="bfloat16",
use_tokenizer_eos=True,
patch_hf_runner=model_utils.internvl_patch_hf_runner,
marks=[large_gpu_mark(min_gb=32)],
),
"llava_next": VLMTestInfo(
models=["llava-hf/llava-v1.6-mistral-7b-hf"],
Expand Down
2 changes: 2 additions & 0 deletions tests/models/multimodal/processing/test_common.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -146,12 +146,14 @@ def _test_processing_correctness(
"facebook/chameleon-7b",
"deepseek-ai/deepseek-vl2-tiny",
"adept/fuyu-8b",
"OpenGVLab/InternVL2-1B",
"llava-hf/llava-1.5-7b-hf",
"llava-hf/llava-v1.6-mistral-7b-hf",
"llava-hf/LLaVA-NeXT-Video-7B-hf",
"llava-hf/llava-onevision-qwen2-0.5b-ov-hf",
"TIGER-Lab/Mantis-8B-siglip-llama3",
"mistral-community/pixtral-12b",
"nvidia/NVLM-D-72B",
"openbmb/MiniCPM-o-2_6",
"openbmb/MiniCPM-V-2_6",
"Qwen/Qwen-VL-Chat",
Expand Down
Loading