[Model] Merged multimodal processor for Paligemma #13584
Conversation
kylehh
commented
Feb 20, 2025
•
kylehh
commented
- Created merged MM processor for Paligemma
- Issue to be solved:
- need newline at prompt (solved)
- paligemma 2 support (solved by adding token)
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels. Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add 🚀 |
|
This pull request has merge conflicts that must be resolved before it can be |
|
Newline issue is solved by overriding |
| tokenizer = self.info.get_tokenizer() | ||
|
|
||
| mm_token_matches = { | ||
| modality: find_token_matches(token_ids, prompt_repls) | ||
| for modality, prompt_repls in mm_prompt_repls.items() | ||
| } | ||
| mm_match_counts = { | ||
| modality: len(matches) | ||
| for modality, matches in mm_token_matches.items() | ||
| } | ||
|
|
||
| # If the search text does not represent a special token, | ||
| # it may have different token IDs in the prompt, because | ||
| # the tokens may go across the boundaries of the search text. | ||
| # ---- | ||
| # e.g. when searching for "foo" in "food", if "food" itself makes | ||
| # up a token, then the token ID of "foo" will not appear at all | ||
| # ---- | ||
| # Since it is inefficient to search for all possible tokenizations | ||
| # of the search text in the prompt, we instead perform string | ||
| # replacement on the decoded token IDs, then encode them back. | ||
| if all( | ||
| mm_match_counts.get(modality, 0) >= item_count | ||
| for modality, item_count in mm_item_counts.items() | ||
| ): # yapf: disable | ||
| token_ids = replace_token_matches( | ||
| token_ids, | ||
| mm_token_matches, | ||
| mm_item_counts, | ||
| ) | ||
|
|
||
| class PaliGemmaMultiModalProjector(nn.Module): | ||
| text = decode_tokens(tokenizer, token_ids) | ||
| matched_repls = { | ||
| modality: [match.prompt_repl for match in token_matches] | ||
| for modality, token_matches in mm_token_matches.items() | ||
| } | ||
| else: | ||
| text = decode_tokens(tokenizer, token_ids) | ||
|
|
||
| mm_text_matches = { | ||
| modality: find_text_matches(text, prompt_repls) | ||
| for modality, prompt_repls in mm_prompt_repls.items() | ||
| } | ||
| text = replace_text_matches( | ||
| text, | ||
| mm_text_matches, | ||
| mm_item_counts, | ||
| ) | ||
|
|
||
| def __init__(self, vision_hidden_size: int, projection_dim: int): | ||
| super().__init__() | ||
| token_ids = encode_tokens(tokenizer, | ||
| text, | ||
| add_special_tokens=False) | ||
| matched_repls = { | ||
| modality: [match.prompt_repl for match in token_matches] | ||
| for modality, token_matches in mm_text_matches.items() | ||
| } | ||
|
|
||
| placeholders = self._find_mm_placeholders( | ||
| matched_repls, | ||
| token_ids, | ||
| mm_item_counts, | ||
| ) |
There was a problem hiding this comment.
Can we simply call super()._apply_prompt_replacements?
|
This pull request has merge conflicts that must be resolved before it can be |
|
Please remember to test this by adding the model to |
