The Serving Stack

Every earlier chapter zoomed in on one mechanism. This one zooms out to the whole path. A production LLM endpoint is a pipeline of seven stages, and almost every latency problem, outage or cost overrun can be traced to one of them.

The life of one request

In timing terms:

• TTFT (time to first token) = gateway + routing + tokenization + queueing in the scheduler + prefill + one network hop back. Queueing is the part that blows up under load.
• ITL (inter-token latency) = one decode step for the whole batch, plus detokenization and streaming, which are usually small.
• Everything in front of the engine should add milliseconds, not hundreds of them. If it does, look for proxies that buffer responses (common with default nginx or ingress settings), which breaks streaming entirely.

See latency metrics for exact definitions.

Choosing an engine

Whatever you pick, pin the exact versions of the engine, model revision, tokenizer and chat template, and the quantized artifacts. A silent change to any of them changes outputs.

The flags that matter

Engines have hundreds of options, but a handful decide memory, concurrency and latency. Build a vllm serve command and watch what each change does:

A practical tuning order:

1. --max-model-len: set it to what your product really needs. A 128K default reserves room you will never use.
2. --gpu-memory-utilization: as high as is stable (0.90–0.95), so KV gets everything left over.
3. --tensor-parallel-size: the smallest value that leaves healthy KV room.
4. --kv-cache-dtype fp8 and --quantization: once you have evals that show quality holds.
5. --max-num-seqs and --max-num-batched-tokens: tune last, against your TTFT and ITL SLOs under realistic load.

Running it on Kubernetes

The engine is just a container, but a very heavy one: a 10 GB image, tens to hundreds of GB of weights, and minutes of initialization. The deployment shape reflects that:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llama-70b
spec:
  replicas: 2
  selector: { matchLabels: { app: llama-70b } }
  template:
    metadata: { labels: { app: llama-70b } }
    spec:
      nodeSelector: { nvidia.com/gpu.product: NVIDIA-H100-80GB-HBM3 }
      containers:
        - name: vllm
          image: vllm/vllm-openai:<pinned-version>
          args: ["meta-llama/Llama-3.1-70B-Instruct",
                 "--tensor-parallel-size", "4",
                 "--max-model-len", "16384",
                 "--gpu-memory-utilization", "0.92"]
          resources:
            limits: { nvidia.com/gpu: 4 }
          volumeMounts:
            - { name: weights, mountPath: /root/.cache/huggingface }   # pre-staged weights
            - { name: shm, mountPath: /dev/shm }                       # needed for TP workers
          startupProbe:   { httpGet: { path: /health, port: 8000 }, periodSeconds: 10, failureThreshold: 90 }
          readinessProbe: { httpGet: { path: /health, port: 8000 }, periodSeconds: 5 }
          livenessProbe:  { httpGet: { path: /health, port: 8000 }, periodSeconds: 15, failureThreshold: 6 }
      volumes:
        - { name: weights, persistentVolumeClaim: { claimName: llama-70b-weights } }
        - { name: shm, emptyDir: { medium: Memory, sizeLimit: 16Gi } }

What makes cold starts slow, and what makes them fast:

Operational rules of thumb:

• Startup probe with a generous budget, so a slow weight load is not mistaken for a crash and restarted in a loop.
• Readiness only after warm-up, so the first real user doesn’t pay for compilation.
• Lenient liveness: an engine at 100% load answers health checks slowly, and killing it under load turns a spike into an outage.
• Treat a TP group (or a multi-node PP group) as one unit: if one rank dies, the whole replica is down.
• Watch GPU health through DCGM: Xid errors, ECC errors, thermal throttling.
• Scale on queue depth and KV usage, not CPU or GPU utilization. See routing & autoscaling.