Benchmarking

The metrics chapter defined what to measure: TTFT, ITL, TPOT, E2E and goodput. This chapter is about producing those numbers so they predict production. A useful benchmark has three parts, and getting any one wrong makes the result meaningless:

• The workload. Prompt and output length distributions, prefix sharing, and the request mix of your traffic, not a default 1024 / 128.
• The load model. How requests arrive: a fixed number in flight, or a rate that ignores how fast the server answers.
• The method. Warm-up, run length, sweep points, and where the client sits.

The output is a single sentence you can plan with: “N req/s at p99 TTFT ≤ X and p99 ITL ≤ Y, for this workload, on this engine version and these flags.” That sentence feeds capacity planning.

Start from the workload shape

Two numbers describe most requests: the input sequence length (ISL) and the output sequence length (OSL). Their ratio decides which phase of the engine does the work. Long inputs stress prefill and TTFT; long outputs stress decode, ITL and KV cache capacity. Their spread matters as much as their median: a p99 prompt ten times the median is what breaks TTFT.

The best source of a workload is your own traffic: sample request logs (lengths and, where allowed, prompts), keep the session structure for multi-turn and agent traffic, and replay it. Synthetic data is fine for sweeps if you match the distribution and the prefix structure, not only the means.

Closed loop vs open loop

Load generators drive the server in one of two ways, and they produce different curves from the same server:

• Closed loop (fixed concurrency). N virtual users each send a request, wait for the full response, then send the next. --max-concurrency 32 in vllm bench serve is a closed loop. When the server slows down, the client slows down with it, so the queue can never exceed N.
• Open loop (fixed request rate). Requests arrive at rate λ, usually as a Poisson process, whether or not earlier requests have finished. --request-rate 8 is an open loop. Real users behave like this: they do not wait for each other.

A closed loop hides queueing. The client stops sending exactly when the server falls behind, so the bursts that build queues in production never happen. This is a version of coordinated omission: the test avoids measuring the moments that would have been worst. Below the knee the two models roughly agree. Near the knee, a Poisson stream at the same average rate has a much worse tail, and past capacity an open-loop queue grows without limit.

Sweeping means running the benchmark at a series of load levels and plotting latency against throughput. A good sweep has many points near the knee and a few beyond it. Practical rules:

• Use concurrency sweeps to find the knee quickly and to compare engines at equal concurrency. Then confirm the capacity with an open-loop run at the rate you plan to serve, ideally with some burstiness.
• vllm bench serve sends every request at time zero when --request-rate is left at its default of inf. That measures a thundering herd, not a service. Always set a rate or a concurrency.
• Read the knee off the p99 curves against your SLO, not off the throughput curve. Throughput keeps rising slowly long after users have started to suffer.

Running it properly

Warm-up. The first requests after start-up pay for kernel compilation, CUDA graph capture, lazy allocation and a cold prefix cache. Send a warm-up phase and exclude it (--num-warmups in vllm bench serve, --warmup-requests in SGLang’s benchmark), and benchmark only after readiness passes.

Duration and sample size. A p99 is decided by the slowest 1% of requests. With 200 requests that is two requests. Run each sweep point long enough for at least a few hundred, ideally 1,000+ requests, and for open-loop tests at least a few minutes so queues reach steady state. Repeat the run before believing a 5–10% difference.

Client-side bottlenecks. The load generator is a program too. A single Python process parsing hundreds of concurrent token streams can run out of CPU, and then it measures its own event loop: ITL rises and throughput plateaus while the GPU is half idle. Watch the client’s CPU, run it on a separate machine close to the server, and spread very high concurrency over several client processes. Check which tokenizer the client uses to count tokens; counting with a different tokenizer than the server shifts every per-token number.

Where to point it. Measure twice, at two places:

The difference between the two is your platform overhead; it should be a few milliseconds of TTFT, not hundreds. Avoid kubectl port-forward for load tests: it tunnels through the API server and becomes the bottleneck long before the GPU does. Through a gateway, remember that routing changes results: a prefix-aware router turns repeated prompts into cache hits that a direct-to-pod test never sees.

The tools

All of these speak the OpenAI-compatible API, stream responses, and report TTFT, ITL and throughput percentiles. They differ in load models, datasets and reporting.

# vLLM: one closed-loop sweep point, with SLO-based goodput
vllm bench serve --backend openai-chat --endpoint /v1/chat/completions \
  --base-url http://llama-8b.serving.svc:8000 --model meta-llama/Llama-3.1-8B-Instruct \
  --dataset-name random --random-input-len 1024 --random-output-len 256 \
  --max-concurrency 32 --num-prompts 1000 --num-warmups 50 \
  --percentile-metrics ttft,tpot,itl,e2el --metric-percentiles 50,90,99 \
  --goodput ttft:500 tpot:50
# ...the same, open loop: replace --max-concurrency with --request-rate 8

# GuideLLM: automatic sweep from synchronous to max throughput
guidellm run --backend kind=openai_http,target=http://llama-8b.serving.svc:8000 \
  --profile kind=sweep \
  --data kind=synthetic_text,prompt_tokens=1024,output_tokens=256 \
  --constraint kind=max_duration,seconds=120

# AIPerf: fixed concurrency, streaming
aiperf profile --model meta-llama/Llama-3.1-8B-Instruct --url http://llama-8b.serving.svc:8000 \
  --endpoint-type chat --streaming --concurrency 32 \
  --synthetic-input-tokens-mean 1024 --output-tokens-mean 256

# SGLang's benchmark, against an SGLang server
python -m sglang.bench_serving --backend sglang --host 127.0.0.1 --port 30000 \
  --dataset-name random --random-input-len 1024 --random-output-len 256 \
  --num-prompts 1000 --request-rate 8

How benchmarks lie

Most misleading benchmarks contain no false numbers. Each choice below is defensible on its own; together they can make the same server look 10–30× better, or worse, than what users get.

The usual suspects, and the question that exposes each:

• Aggregate tok/s with no latency. At what p99 TTFT and ITL? Peak throughput is measured where users have already left.
• Counting input tokens. Output or total tokens? vllm bench serve prints both output and total token throughput; prompt-heavy workloads make “total” several times larger.
• ignore_eos with fixed lengths. Uniform outputs keep batches neat and hide models that ramble or stop early, which is exactly what a quantized model can do.
• Random prompts have no shared prefixes, so prefix caching never helps and the benchmark understates production. Identical prompts hit the cache every time and overstate it. Use real prompts or a dataset with your prefix structure (--random-prefix-len, prefix_repetition).
• Mismatched engine defaults. Comparing engines “out of the box” compares their defaults: prefix caching, chunked-prefill budget, max batched sequences, CUDA graphs, speculative decoding, KV-cache dtype, even the chat template. Pin and list every flag on both sides.
• Averages. Means hide the tail; report p50, p90 and p99.
• Ignoring cold start. No warm-up penalises engines with long compilation; a warm-only benchmark hides the cold start users hit after every scale-up. Measure both, separately.

A reporting template

Every benchmark result should travel with its context, or nobody can reproduce or compare it: