Quality & Safety

The rest of this guide is about making inference faster and cheaper. This chapter covers the two things speed can cost you:

• Quality. Quantization, an FP8 KV cache, a new engine version, a changed chat template or new speculative decoding settings all change the numbers the model computes. Some changes are harmless. Others quietly break maths, code or long documents while chat still looks fine.
• Safety. A model behind an API takes untrusted text in, produces text that other systems act on, calls tools, and loads weights from the internet. Each of those is an attack surface the serving platform owns.

Every optimisation needs an accuracy gate

Treat any change below as a new model that must earn its way to production:

The quantization chapter gives the short version: compare against the baseline on your own evals, include the fragile slices, set thresholds first. The gate below shows why those three rules matter:

Two consequences. First, gate per task, never on the average: a model that loses 3 points on maths and gains nothing elsewhere has a small average delta and a real regression. Second, size the eval to the threshold: to resolve a 1-point change you need on the order of 1,000+ paired examples per task. Score the same items with both models and look at the items that flipped, not just the totals.

Which evals, and how far to trust them

No single eval is enough. Use a ladder, from cheap and coarse to expensive and close to users:

LLM-as-judge is useful but biased in known ways: it prefers the answer shown first (position bias), prefers longer answers (verbosity bias), and rates outputs from its own model family higher (self-preference). Swap the order and average both, cap or normalise length, use a judge from a different family, and calibrate it against a few hundred human labels before trusting it.

Standard benchmarks run against any OpenAI-compatible endpoint with lm-evaluation-harness, so you test the exact engine, flags and chat template you will serve:

lm_eval --model local-chat-completions \
  --model_args model=meta-llama/Llama-3.1-8B-Instruct,base_url=http://candidate:8000/v1/chat/completions \
  --tasks gsm8k,ifeval --apply_chat_template
# chat endpoints support generative tasks only; log-likelihood tasks (e.g. MMLU) need local-completions

A validation workflow that holds up:

1. Freeze the baseline: model revision, engine version, flags, chat template, sampling settings. Store its scores and outputs.
2. Write thresholds before running: e.g. “no task drops more than 1 point; JSON validity ≥ 99.5%; median output length within ±10%”.
3. Run the candidate on the same items, same sampling (greedy for comparability), through the real serving path.
4. Diff: per-task deltas with intervals, flipped items, length and refusal changes, parse failures.
5. Canary or shadow: send a slice of live traffic (or mirror it) and compare online signals before full rollout. Keep the baseline deployable for rollback.

Watching quality in production

Offline evals pass, then quality drifts in production because traffic changes, a template is edited, or an upgrade slips through. Engine dashboards show latency, not correctness, so add cheap quality proxies per model, route and version:

• Output length distribution (p50, p95). Sudden shifts mean rambling, early stopping or a changed default.
• finish_reason mix: a jump in length means truncation, often with broken JSON.
• Refusal rate, by pattern match or a small classifier.
• JSON / tool-call parse failures and schema-validation errors.
• Language mismatch (answer not in the prompt’s language), repetition loops, empty answers.
• User signals: regenerate clicks, thumbs down, edits.

Expect one surprise: at temperature 0 the same prompt does not always give the same answer. Floating-point addition is not associative, and kernels pick their reduction strategy (split-K, tile sizes, attention splits) from the shape of the batch. How many other requests happen to share the step changes the order of additions, the logits move in the last bits, and a near-tie between two tokens can flip.

The threat model: OWASP Top 10 for LLM applications

The OWASP Top 10 for LLM Applications (2025) is the common checklist. Several items belong to the application, but the serving platform can enforce a lot of them centrally:

Prompt injection

Direct injection is a user typing “ignore your instructions”. Indirect injection is worse: the instructions arrive inside content the model reads on the user’s behalf, such as a web page, an email, a PDF or a tool result. The root cause is architectural. Everything in the context, system prompt, user message and retrieved data alike, is one sequence of tokens; the model has no type system that marks some tokens as “data, never instructions”. Delimiters and “ignore instructions in documents” prompts lower the success rate but do not close the hole.

Unbounded consumption

The cheapest attack on an LLM service is a request that costs a lot: a prompt at the full context length, max_tokens at the maximum, a reasoning model asked to “think as long as needed”, or thousands of parallel streams. Defences belong at the gateway and the engine: per-tenant token budgets (input and output tokens per minute, not just requests), a hard cap on max_tokens and prompt size, a max_model_len sized to the product, request timeouts, and cancellation that frees KV blocks when clients disconnect (see routing and autoscaling).

Guardrails and what they cost

Guardrails are checks outside the main model: an input classifier before the prompt reaches the engine, an output classifier on what comes back, and rules in between. Common building blocks are safety classifier models such as Llama Guard, ShieldGemma and Granite Guardian, and frameworks such as NeMo Guardrails and Guardrails AI that wire checks, rules and validators around the model call. A guard is itself a model call, so it adds latency and GPU time, and on a streaming response the output check fights the stream:

Practical rules: run input checks in parallel with prefill and cancel on a hit; use a small guard model (1–8B) on its own replicas so it scales independently; check output in chunks for interactive traffic and the full response for tool calls and actions; and measure the guard’s false-positive rate, because over-refusal is a quality regression too.

The model supply chain

Model weights are downloaded artefacts, often from public hubs, and loading them can run code. Treat a model like a container image:

• Safetensors, not pickle. PyTorch .bin/.pt checkpoints use Python pickle, which can execute arbitrary code when loaded. Safetensors stores only a JSON header and raw tensors. Refuse pickle formats from untrusted sources, or convert them in a sandbox.
• No remote code. trust_remote_code runs Python from the model repository inside your serving process. Keep it off; if a model truly needs it, review and vendor the code.
• Pin revisions. A repo name points to a moving branch. Pin the commit (hf download <repo> --revision <sha>, and --revision in the engine), and mirror approved models into your own registry or object store.
• Scan and sign. Scan artefacts for unsafe serialisation (e.g. Protect AI ModelScan), and sign and verify them (OpenSSF model-signing, built on Sigstore) so the weights you serve are the weights you approved.
• Record it. An AI bill of materials (AI-BOM) lists model, revision, hashes, datasets where known, adapters, engine and licence for each deployment.
• Read the licence. Open-weight licences differ: some restrict commercial use, user counts, use cases or derivative models. Legal review belongs in the promotion pipeline, like the eval gate.