Observability & Troubleshooting

The metrics chapter defined what users feel: TTFT, ITL, goodput and the dashboard panels that show them. This chapter goes one level lower. A latency SLO alert says that something is wrong. Finding out where means looking through six layers, each with its own signals and failure modes:

Two rules make this tractable. First, collect every layer with the same labels (node, GPU UUID, pod, model) so a slow request can be followed down to a physical GPU. Second, start from the symptom, not from whichever dashboard is open.

GPU signals from DCGM

On Kubernetes the GPU Operator runs dcgm-exporter as a DaemonSet. It scrapes NVIDIA’s Data Center GPU Manager and exposes Prometheus metrics labelled with GPU, node and, when pod mapping is on, the pod and namespace using the GPU. The default counter list covers the essentials. Some of the most useful profiling fields ship commented out and have to be enabled in a custom counters CSV.

The single most misread number is the first utilisation column. Play with the kernel pattern of a decode step and watch what each counter says:

GPU_UTIL is a time-based “was anything running?” flag. It cannot tell one warp on one SM from all 132 SMs saturating their tensor cores. For capacity questions, use the profiling metrics together with the engine’s own numbers: running batch, KV usage and tokens per second. The roofline explains why decode lives at high DRAM activity and low tensor activity, and why that is fine.

Clocks, power and heat

A GPU slower than its neighbours is often running below its maximum clocks. The driver records why in a set of clock event reasons (older drivers and tools call them “throttle reasons”):

Two lessons carry over to real fleets:

• Prefill feels clocks, decode mostly does not. A power-capped GPU shows up as worse TTFT under long prompts, while ITL stays nearly flat. Decode waits on HBM, and memory clocks are not what the power cap reduces first.
• The reason matters more than the clock. gpu_idle and sw_power_cap on a capped GPU are normal behaviour. hw_thermal_slowdown on one node and not its neighbours points to a fan, a blocked inlet or a rack hot spot. hw_power_brake_slowdown points to the facility or the PSU. dcgm-exporter can count clock events per reason through its custom collector; enable it if you want alerts on reasons rather than raw clocks.

Xid errors and memory health

When the driver detects a GPU error, it writes an Xid line to the kernel log (NVRM: Xid (PCI:0000:…): 79, …). The number tells you whether to look at your code or call the hardware vendor:

Modern data-centre GPUs (A100 and later) handle HBM errors by row remapping: a row with uncorrectable errors is replaced by a spare. The remap is recorded immediately (Xid 63) but only takes effect after a GPU reset. If the spares run out or the remap fails (Xid 64), the GPU should leave service. That gives a simple policy: alert on DCGM_FI_DEV_ROW_REMAP_FAILURE, track UNCORRECTABLE_REMAPPED_ROWS per GPU, and schedule a drain and reset when remaps are pending.

Engine signals in a triage

The engine metrics from the metrics chapter do the second half of the diagnosis. These are the ones that separate causes:

• vllm:num_requests_waiting above zero for a sustained period means queueing, so TTFT will rise. Compare it across replicas: if one replica queues while others idle, the problem is routing, not capacity.
• vllm:kv_cache_usage_perc near 1 means concurrency is limited by KV memory, not by compute.
• vllm:num_preemptions_total increasing means running sequences are being evicted and recomputed. This shows up as ITL spikes and wasted GPU time. See the scheduler chapter.
• Prefix hit rate: the rate of vllm:prefix_cache_hits_total divided by the rate of vllm:prefix_cache_queries_total. A sudden drop means more prefill per request: new replicas, a routing change, or a prompt that stopped being byte-identical.

Symptom first: a triage path

Pick the symptom you see, then answer each check in turn. Each step names the command or metric that answers it:

The commands that come up again and again:

# Live per-GPU view: clocks, power, temperature, why clocks are reduced
nvidia-smi --query-gpu=index,name,temperature.gpu,power.draw,power.limit,clocks.sm,clocks.max.sm,utilization.gpu,memory.used,clocks_event_reasons.active --format=csv -l 5
nvidia-smi -q -d PERFORMANCE          # clock event reasons, human-readable
nvidia-smi -q -d ECC,ROW_REMAPPER     # ECC counters and remapped rows
nvidia-smi topo -m                    # NV# = NVLink, PIX/PXB/PHB/SYS = PCIe paths between GPUs and NICs
nvidia-smi nvlink -s                  # per-link status
dmesg -T | grep -i xid                # driver-reported GPU errors on the node
dcgmi diag -r 1                       # seconds: deployment / software checks
dcgmi diag -r 3                       # ~15 min: hardware tests, run on a drained node

And a starting set of alerts. Label names depend on your exporter configuration:

- alert: GpuXidError            # Xid gauge holds the last code; alert when it changes
  expr: changes(DCGM_FI_DEV_XID_ERRORS[10m]) > 0
- alert: GpuRowRemapFailure     # take the GPU out of service
  expr: DCGM_FI_DEV_ROW_REMAP_FAILURE > 0
- alert: GpuUncorrectableRemap  # schedule drain + reset
  expr: increase(DCGM_FI_DEV_UNCORRECTABLE_REMAPPED_ROWS[1h]) > 0
- alert: GpuHot
  expr: DCGM_FI_DEV_GPU_TEMP > 85
  for: 10m
- alert: GpuAllocatedButIdle    # paying for a GPU nobody uses
  expr: avg_over_time(DCGM_FI_DEV_GPU_UTIL{pod!=""}[1h]) < 1
- alert: EngineQueueing
  expr: max by (pod) (vllm:num_requests_waiting) > 0
  for: 5m
- alert: EnginePreempting
  expr: rate(vllm:num_preemptions_total[5m]) > 0
  for: 10m