Multimodal Serving

A vision-language model (VLM) is a normal LLM with two extra parts in front of it: a vision encoder that reads pixels, and a small projector that maps the encoder’s outputs into the LLM’s embedding space. After that, the LLM cannot tell an image from text. Image positions are just more tokens in the prompt.

That one fact explains most of multimodal serving:

• Images are prompt tokens. A single photo can be 256 to over 16,000 tokens depending on the model and its resolution settings. They go through prefill, take KV cache and count against the context like any text.
• There is extra work before prefill. The image has to be uploaded, decoded and resized on the CPU, then run through the encoder on the GPU. All of it lands in TTFT.
• Requests get lumpy. One request with eight scanned pages is a very long prompt, with all the interference a long prompt brings.

How an image becomes tokens

The standard recipe, used with variations by Qwen-VL, LLaVA, InternVL and most open VLMs:

1. Resize the image to a size the encoder accepts.
2. Cut it into patches, usually 14×14 or 16×16 pixels. Each patch is one input position for a ViT (vision transformer), which runs full transformer layers over the patch sequence.
3. Merge and project. Neighbouring patch outputs are usually merged (2×2 → 1 is common) and an MLP projects them to the LLM’s hidden size. Each merged vector becomes one LLM token.
4. Splice the image tokens into the text sequence where the image placeholder sits in the chat template.

So the token count depends on the resolution after resizing, and each model family chooses a different scheme for that resize:

The cost lands in TTFT

Before the first output token, an image request does four things the text path does not, and two of them do not happen on the GPU:

What each stage teaches:

• Upload. Base64 inflates the bytes by 4/3, and a 12 MP JPEG is several megabytes. Gateways and proxies have body-size limits (nginx’s default is 1 MB), and API servers buffer whole requests. Resize on the client to roughly what the model will use, or pass a URL the server can fetch. URL fetching then needs an allow-list (vLLM has --allowed-media-domains), or it becomes a server-side request forgery path.
• CPU preprocessing. Decoding and resizing run on the API server’s CPU. Under load, they compete with tokenization and detokenization, and the GPU waits. Scale out API server processes, give the pod enough cores, and watch CPU time per request as well as GPU metrics.
• Encoder. The ViT is small next to the LLM but runs over every patch (4× the LLM tokens with a 2×2 merge). With tensor parallelism, splitting a small encoder across GPUs is often wasteful; vLLM’s --mm-encoder-tp-mode data runs it data-parallel instead, for supported models.
• Prefill. Image tokens prefill like text. A request with 8 images can be a 10,000-token prompt, so chunked prefill matters as much as for long documents, and an image-heavy request can stall everyone else’s decode without it.

Caching images

Users send the same images again: the same document asked about five times, a screenshot re-sent with every turn of a chat, a product photo shared by many requests. Engines avoid repeating the work at several levels:

The prefix cache is the big win, and it follows the usual rule: matching is positional. An image after the same system prompt in the same position hits; the same image after a different preamble misses at the first differing block, so put shared images and documents before the per-request text.

Limits and admission

Multimodal requests have more ways to be huge, so engines add guard rails:

• Items per prompt. --limit-mm-per-prompt caps how many images, videos or audio clips a prompt may contain, per modality (for example --limit-mm-per-prompt.image 4, or a JSON map). Set a modality to 0 to disable it. The engine plans its startup memory profiling around the worst case these limits allow, so overly generous limits waste KV memory on a case that never arrives.
• Pixels per item. Processor arguments such as max_pixels bound tokens per image; frame counts and frame sizes bound video.
• Context. All of it must fit in max_model_len together with the text and the output.
• Payload. Gateway body limits, request timeouts and per-tenant rate limits should count image bytes and image tokens, not just requests.

Video and audio: the budget explodes

Video is a stack of images. A model samples frames at some rate, encodes each one (often merging pairs of consecutive frames, as Qwen2.5-VL does), and every frame costs tokens. Audio encoders produce a steady stream of tokens per second of sound. The multiplication gets out of hand quickly:

For long media, the practical patterns are the same as for long documents: sample less, shrink frames, cut into segments and process them separately (in parallel, on several replicas), and summarise into text that later turns can reuse cheaply.

Splitting the encoder off

Because the encoder is a separate model with a different compute profile, it can run on separate workers, the same idea as prefill/decode disaggregation applied one stage earlier. vLLM’s disaggregated encoder mode runs encoder instances that produce image embeddings and ship them to prefill/decode instances through an encoder-cache connector (--ec-transfer-config, with producer and consumer roles). Text-only requests skip the encoder pool entirely. It is newer and less mature than P/D disaggregation, but it addresses the same problem: a burst of image-heavy requests should not slow everyone’s decode.