Embeddings & Rerankers

This chapter is about serving models whose output is a vector or a score: the embedding models behind semantic search and RAG, and the rerankers that clean up their results. (For the token embeddings and RoPE inside every LLM, see Embeddings & Positions.)

These models are the opposite of chat from the engine’s point of view:

• Prefill only. One forward pass over the input, then done. There is no decode loop, no streaming and no time-per-output-token.
• Nothing to keep. The KV cache lives only for the duration of the pass. No KV cache is held between requests, so memory goes to batch activations instead of sequences waiting for their next token.
• Compute-bound. A whole sequence is processed at once, so arithmetic intensity is high and the GPU’s FLOPS, not its bandwidth, set the roofline.
• Measured differently. Throughput is sequences/s or tokens/s; latency is a single number per request, with no TTFT/ITL split.

Three ways to compare text

A bi-encoder must squeeze a document’s meaning into one vector before it knows the question. A cross-encoder lets every query token attend to every document token, so it is much more accurate, but it cannot pre-compute anything. That is why real pipelines use both: a bi-encoder to find 100 candidates in a billion, a cross-encoder to pick the best 5.

From token vectors to one vector: pooling

A transformer outputs one hidden vector per token. An embedding model reduces them to one vector with a pooling rule fixed at training time:

• CLS pooling: take the first token’s state. BERT-style encoders trained this way use it as a summary.
• Mean pooling: average all real tokens. Common for sentence-transformer models.
• Last-token pooling: take the final token’s state. Used by decoder (LLM-based) embedding models, because under causal attention only the last token has seen the whole text.

Then the vector is usually L2-normalised, so that a plain dot product equals cosine similarity.

Matryoshka (MRL) models are trained so that the first d dimensions are themselves a usable embedding. You can store 256 of 1,024 dimensions and lose a little recall for 4× less memory. In vLLM the OpenAI-style dimensions request field selects the size, and it is rejected for models that were not trained this way.

Batching without paying for padding

An embedding server sees texts of wildly different lengths: 10-token queries, 300-token chunks, the occasional 4,000-token document. Classic encoder batching pads every sequence to the longest in the batch, and every padding position goes through every matmul.

The practical rules:

• Batch by tokens, not by count. Engines with variable-length attention (vLLM, TEI) pack sequences up to a token budget, the same idea as continuous batching with a chunked-prefill-style budget.
• Sort offline jobs by length. When indexing a corpus, bucketing by length gets most of the benefit even on a padded runtime.
• Respect max sequence length. Text past the model’s limit (often 512 tokens for BERT-style encoders) is silently truncated. Chunk documents to fit; do not rely on truncation.
• Attention is quadratic. For long-context embedding models, a few long documents dominate the batch’s FLOPs, so cap chunk size even when the model allows 8K or more.

Serving it: engines and endpoints

Two engines cover most deployments:

• vLLM runs embedding, reranking and classification models with the pooling runner (--runner pooling, usually detected automatically). It exposes /v1/embeddings, /pooling, /classify, /score and /rerank (also /v1/rerank). This suits LLM-based embedding models and lets you reuse the same stack as your generative models.
• Text Embeddings Inference (TEI) from Hugging Face is purpose-built for encoders and rerankers, with token-based dynamic batching. It serves /embed, /rerank, /predict and an OpenAI-compatible /v1/embeddings.

Both are prefill-only workloads, so the roofline points to FLOPS per dollar, not HBM bandwidth. A 300M-parameter encoder needs only a few GB of memory, which leaves two sensible shapes. For bulk indexing, where you can keep any GPU saturated, cost per embedding is roughly $/hour ÷ FLOPS, and large GPUs are often cheapest. For live query traffic, a few hundred requests per second cannot keep an H100 busy, so you pay per hour, not per embedding. A small GPU such as an L4 (24 GB) or L40S (48 GB, no NVLink), neither of which supports MIG, or a single MIG slice such as an H100 1g.10gb (see GPU sharing) is the better fit.

Storage is often the bigger bill. 100M documents × 1,024 dims × 4 bytes = 410 GB of raw float32 vectors, before any index. Matryoshka truncation to 256 dims, int8 quantization (4×) or binary quantization (32×) bring that down to something that fits in RAM, usually with a second re-scoring pass over the top candidates using full-precision vectors or a reranker to recover recall.

Rerankers and the RAG latency budget

At query time, a RAG request runs a chain: embed the query, search the index, rerank the top-k, then prefill the LLM with the best chunks and generate. Each stage spends part of the user’s TTFT budget.

What the budget shows:

• Query embedding and vector search are cheap, a few milliseconds each. Approximate nearest-neighbour indexes grow roughly with log N in time, and linearly in memory.
• The reranker is the knob. Its cost scales with k × chunk length × reranker size. Rerank 50, not 500. A good reranker also lets you send fewer chunks to the LLM, and LLM prefill time falls with prompt length.
• Index time and query time are different workloads. Bulk indexing wants maximum throughput: huge length-sorted batches, cheap GPUs, run offline. Query-time embedding and reranking want low latency with small batches. Run them as separate deployments, or at least with separate priorities, so a re-indexing job never queues behind or in front of live queries.