[Feature] Enhanced support/structure for Multi-modal models

[tp-nan]

Checklist

• 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 2. Please use English, otherwise it will be closed.

Motivation

In vllm, the framework can accept either image or image-embedding. See vllm [Feature] Add vision language model support. #3042 and vllm-llava impl.

Embedding an image requires fixed predictable compute and is easy to batch and run in a separate framework(for instance, tensorrt-based serving framework). See discussion in vllm-project/vllm#307 (comment)

Ideally, it is necessary to maintain the infrastructure to overlap (image (gpu) preprocessing + inference) and (llm inference) within the same process (avoiding the need for nvidia MPS)

Related resources

No response

[tp-nan]

Sorry, duplicated from #745