72 terms

Glossary

Short definitions, each linked to the chapter that teaches the idea. Inside chapters, terms with a dotted underline show the same definition on hover or tap.

Part IInside the model

token: The unit a model reads and writes: a word, part of a word or a symbol, mapped to an integer ID from a fixed vocabulary. Tokenization →
BPE · byte-pair encoding: Byte-pair encoding: a tokenizer built by repeatedly merging the most frequent adjacent pair of symbols, so common strings become single tokens. Tokenization →
chat template: The model-specific wrapper of special tokens (roles, turn separators) that turns a list of messages into the single token sequence the model was trained on. Tokenization →
embedding: A learned vector for each token ID. The first layer of the model is a table lookup from ID to embedding. Embeddings & RoPE →
RoPE · rotary position embedding: Rotary position embedding: rotates query and key vectors by an angle proportional to position, so attention scores depend on relative distance. Embeddings & RoPE →
attention: The operation that lets each token read from earlier tokens: softmax(QKᵀ/√d)·V, with a causal mask during generation. Attention →
causal mask: Sets attention scores to −∞ for future positions, so each token only sees itself and earlier tokens. It is what makes caching K and V valid. Attention →
GQA · grouped-query attention: Grouped-query attention: several query heads share one K/V head, cutting the KV cache by the group size (e.g. 4–8×) at little quality cost. Attention →
MQA: Multi-query attention: all query heads share a single K/V head. The smallest KV cache, with some quality loss. Attention →
MLA · multi-head latent attention: Multi-head latent attention (DeepSeek-V2/V3): caches one compressed latent vector per token per layer and reconstructs per-head K and V from it. Attention →
residual stream: The hidden vector that flows through every layer; each attention and MLP block adds its output to it rather than replacing it. Transformer layer →
logits: The raw, unnormalised score the model assigns to every vocabulary entry for the next token. Softmax turns them into probabilities. Sampling →
temperature: Divides the logits before softmax. Below 1 sharpens the distribution toward the top token; above 1 flattens it. As T → 0 it becomes greedy. Sampling →
top-p · nucleus sampling: Nucleus sampling: keep the smallest set of tokens whose probabilities sum to at least p, renormalise, and sample from them. Sampling →
top-k: Keep only the k most likely tokens, renormalise, and sample from them. Sampling →

Part IIOne request, one GPU

HBM · high-bandwidth memory: High-bandwidth memory: the stacked DRAM next to the GPU die that holds weights and the KV cache. Its bandwidth (TB/s) caps decode speed. GPU & memory →
SRAM: Fast on-chip memory (registers, shared memory, L1) inside each SM. Tiny, but an order of magnitude faster than HBM. GPU & memory →
tensor core · tensor cores: Specialised GPU units that do small matrix multiplies per clock. They supply nearly all of a GPU’s headline FLOPs. GPU & memory →
SM · streaming multiprocessor: Streaming multiprocessor: one of the ~100–150 independent compute blocks on an NVIDIA GPU, each with its own tensor cores, registers and shared memory. GPU & memory →
prefill: The first phase of a request: process the whole prompt in one parallel pass and fill the KV cache. Compute-bound. Prefill vs decode →
decode: The second phase: generate one token per forward pass, re-reading the weights and KV cache each step. Memory-bandwidth-bound. Prefill vs decode →
arithmetic intensity: FLOPs performed per byte moved from memory. Below the GPU’s ridge point a kernel is memory-bound; above it, compute-bound. Roofline →
roofline: A plot of attainable FLOP/s against arithmetic intensity: a bandwidth slope that bends into a flat compute ceiling at the ridge point. Roofline →
memory-bound: Limited by how fast data can be read from memory, not by math. Decode at small batch is the classic case. Roofline →
compute-bound: Limited by how fast the tensor cores can do math. Prefill of long prompts is the classic case. Roofline →
KV cache: The keys and values of every past token, kept in GPU memory so each decode step only computes K and V for the new token. KV cache →
FlashAttention: An attention kernel that tiles Q, K and V through on-chip SRAM with an online softmax, never writing the n×n score matrix to HBM. FlashAttention →
online softmax: Computes softmax in one streaming pass by keeping a running max and running sum, rescaling earlier partial results as new blocks arrive. FlashAttention →
kernel fusion: Combining several GPU operations into one kernel so intermediates stay on-chip instead of round-tripping through HBM. Kernels & CUDA graphs →
CUDA graph · CUDA graphs: A recorded sequence of kernel launches that can be replayed with a single CPU call, removing per-kernel launch overhead from tiny decode steps. Kernels & CUDA graphs →

Part IIIMany requests, one engine

continuous batching · iteration-level batching: Scheduling at every decode step: finished sequences leave and new ones join the running batch immediately, instead of waiting for the whole batch. Batching →
PagedAttention: Stores the KV cache in fixed-size blocks mapped through a per-sequence block table, like virtual memory, so it needs no contiguous reservation. PagedAttention →
block table: The per-sequence map from logical KV block number to physical block in GPU memory, used by PagedAttention. PagedAttention →
prefix caching: Reusing the KV blocks of an identical earlier prompt prefix (system prompt, shared document) so that part is not prefilled again. Prefix caching →
chunked prefill: Splitting a long prompt’s prefill into chunks that share each step with ongoing decodes, under a per-step token budget. Chunked prefill →
preemption: Evicting a running request when the KV cache runs out; its KV is later recomputed (or swapped back) when there is room. Scheduler →

Part IVFaster & cheaper

speculative decoding: A cheap drafter proposes several tokens, and the target model verifies them all in one forward pass, keeping the longest accepted prefix. The output distribution is unchanged. Speculative decoding →
acceptance rate: In speculative decoding, the probability that the target model accepts a drafted token. It sets how many tokens each verify pass yields. Speculative decoding →
quantization: Storing weights, activations or KV in fewer bits (FP8, INT8, INT4, FP4) with scale factors, to cut memory and bytes moved. Quantization →
FP8: 8-bit floating point (E4M3 or E5M2). Halves memory versus BF16 and doubles tensor-core throughput on Hopper and Blackwell. Quantization →
BF16 · bfloat16: 16-bit floating point with FP32’s 8-bit exponent and a 7-bit mantissa. The default precision for serving weights. Quantization →
constrained decoding · grammar-constrained decoding: Masking the logits at every step so only tokens allowed by a grammar or JSON schema can be sampled. Structured output →
LoRA: Low-rank adaptation: a fine-tune stored as two small matrices per layer (rank r). Many adapters can share one base model in the same batch. Multi-LoRA →

Part VScaling out

NVLink: NVIDIA’s high-bandwidth GPU-to-GPU link inside a server (H100: 900 GB/s total, 450 GB/s each way). Interconnects →
all-reduce: A collective in which every GPU contributes a tensor and every GPU receives the sum. Tensor parallelism runs two per layer. Interconnects →
tensor parallelism · TP: Splitting each weight matrix across GPUs so they all work on every layer together, joined by all-reduces. Cuts per-GPU memory and latency. TP / PP / DP →
pipeline parallelism · PP: Giving each GPU a contiguous slice of layers; activations flow GPU to GPU. Little communication, but adds pipeline bubbles. TP / PP / DP →
data parallelism: Running independent full replicas of the model and splitting requests between them. TP / PP / DP →
MoE · mixture of experts, Mixture of Experts: Mixture of experts: each layer has many expert MLPs and a router that sends each token to only the top-k, so active parameters ≪ total. MoE & expert parallel →
expert parallelism · EP: Placing different experts on different GPUs; tokens travel to their experts and back with all-to-all communication. MoE & expert parallel →
disaggregation · disaggregated serving, P/D disaggregation: Running prefill and decode on separate GPU pools sized independently, and transferring the KV cache between them. P/D disaggregation →

Part VIProduction

TTFT · time to first token: Time to first token: from request arrival to the first streamed token. Queueing + prefill + one decode step. Metrics & goodput →
TPOT · time per output token: Time per output token: average gap between tokens after the first, (E2E − TTFT) / (output tokens − 1). Metrics & goodput →
ITL · inter-token latency: Inter-token latency: each individual gap between consecutive streamed tokens. Its tail (p99) shows stalls that TPOT averages away. Metrics & goodput →
goodput: Throughput counting only requests that met their latency SLOs. The number worth maximising. Metrics & goodput →
SLO · service-level objective: Service-level objective: a latency target such as “p99 TTFT < 500 ms”. Metrics & goodput →
Little's law · Little’s law: In steady state, requests in flight = arrival rate × time in system. Links concurrency, throughput and latency. Metrics & goodput →
KV-aware routing · cache-aware routing, prefix-aware routing: Sending a request to the replica that already holds its prefix in cache, instead of round-robin. Routing & autoscaling →
cold start: The delay before a new replica can serve: pull the image, load weights to HBM, compile and capture CUDA graphs. Routing & autoscaling →
vLLM: A widely used open-source inference engine that introduced PagedAttention. Serving stack →
SGLang: An open-source inference engine known for RadixAttention prefix caching and fast structured output. Serving stack →

Part VIIBeyond chat

reasoning model · reasoning models: A model trained to emit a long hidden chain of thought before answering, so output tokens (and KV) grow by 10–100×. Reasoning & agents →
pooling: Collapsing a sequence of token vectors into one embedding (mean, CLS token or last token). Embeddings & rerankers →
reranker: A cross-encoder that scores each (query, document) pair jointly; slower than embeddings but more accurate for ordering search results. Embeddings & rerankers →

Part VIIIRunning on Kubernetes

DRA · Dynamic Resource Allocation: Dynamic Resource Allocation: the Kubernetes API for requesting devices by attributes through ResourceClaims, replacing simple GPU counts. GPUs on Kubernetes →
device plugin: The Kubernetes component that advertises GPUs on a node as an extended resource (nvidia.com/gpu) and wires them into containers. GPUs on Kubernetes →
MIG · Multi-Instance GPU: Multi-Instance GPU: hardware partitioning of one GPU into isolated slices, each with its own SMs, L2 and memory. GPU sharing →
MPS: Multi-Process Service: lets several processes run kernels on one GPU concurrently, sharing memory and SMs without hardware isolation. GPU sharing →
time-slicing: Sharing a GPU by giving each process turns. Simple, but no memory isolation and context-switch overhead. GPU sharing →
safetensors: A simple, safe weight file format: a JSON header of tensor names, shapes and offsets followed by raw bytes, so it can be memory-mapped. Weights & loading →
DCGM: NVIDIA Data Center GPU Manager: the agent and exporter behind GPU metrics such as utilisation, memory, power, clocks and errors. Observability →
Xid · Xid error, Xid errors: A numbered error code the NVIDIA driver writes to the kernel log, e.g. 79 = GPU fell off the bus, 48 = double-bit ECC error. Observability →