Learning Paths

You will be able to: Explain what happens to a prompt, why decode is slow, what the KV cache costs and how engines batch requests.

1. 01Text → TokensBPE merges, token IDs, chat templates and why token counts matter.8 min
2. 03AttentionQ·Kᵀ, causal masks, softmax, and MHA vs GQA vs MQA.6 min
3. 07Prefill vs DecodeThe two phases of every request and why they behave so differently.7 min
4. 09The KV CacheWhy it exists, how big it gets, and why it caps concurrency.7 min
5. 12BatchingStatic vs dynamic vs continuous (iteration-level) batching.7 min
6. 13PagedAttentionVirtual memory for the KV cache: blocks, block tables, copy-on-write.8 min
7. 25Latency Metrics & GoodputTTFT, ITL, TPOT, E2E, p99, Little’s law and the goodput knee.8 min
8. 28The Serving StackAPI → gateway → engine → GPU: the full request path, and the engines.7 min

You will be able to: Reason from first principles about memory, bandwidth and scheduling, and answer the usual follow-up questions.

1. 04The Transformer LayerOne layer end-to-end: norms, attention, MLP, residuals, params & FLOPs.5 min
2. 06Anatomy of a GPUHBM, L2, SRAM, tensor cores — and why bandwidth rules inference.8 min
3. 07Prefill vs DecodeThe two phases of every request and why they behave so differently.7 min
4. 08The RooflineArithmetic intensity: when you are memory-bound vs compute-bound.6 min
5. 09The KV CacheWhy it exists, how big it gets, and why it caps concurrency.7 min
6. 12BatchingStatic vs dynamic vs continuous (iteration-level) batching.7 min
7. 13PagedAttentionVirtual memory for the KV cache: blocks, block tables, copy-on-write.8 min
8. 14Prefix CachingHashed blocks, radix trees and LRU eviction to skip repeated prefill.7 min
9. 16Scheduler & PreemptionWaiting / running queues, preemption, recompute vs swap, KV offload.7 min
10. 17Speculative DecodingDraft cheaply, verify in one pass: draft models, EAGLE, Medusa, n-gram.7 min
11. 18QuantizationFP8, INT8, INT4, FP4: bit layouts, scales and what each speeds up.9 min
12. 22ParallelismTensor, pipeline, data and context parallelism and their communication.8 min
13. 25Latency Metrics & GoodputTTFT, ITL, TPOT, E2E, p99, Little’s law and the goodput knee.8 min
14. 29Capacity & CostFrom model + SLO + traffic to GPU count and cost per million tokens.8 min

You will be able to: Size a deployment, get GPUs and weights into pods, pick scaling signals and debug a slow or failing replica.

1. 07Prefill vs DecodeThe two phases of every request and why they behave so differently.7 min
2. 09The KV CacheWhy it exists, how big it gets, and why it caps concurrency.7 min
3. 12BatchingStatic vs dynamic vs continuous (iteration-level) batching.7 min
4. 25Latency Metrics & GoodputTTFT, ITL, TPOT, E2E, p99, Little’s law and the goodput knee.8 min
5. 26BenchmarkingWorkload shapes, concurrency sweeps, the tools, and how benchmarks lie.13 min
6. 28The Serving StackAPI → gateway → engine → GPU: the full request path, and the engines.7 min
7. 27Routing & AutoscalingKV-aware routing, the right scaling signals, cold starts.9 min
8. 29Capacity & CostFrom model + SLO + traffic to GPU count and cost per million tokens.8 min
9. 34GPUs on KubernetesGPU Operator, device plugin, CDI, node labels and DRA claims.12 min
10. 35Sharing a GPUTime-slicing, MPS, MIG and vGPU: isolation vs utilisation.11 min
11. 36Model Weights & LoadingFile formats, where weights live, and how fast they reach HBM.12 min
12. 37Observability & TroubleshootingDCGM metrics, Xid errors and a symptom-first triage path.11 min

You will be able to: Read a roofline, find the bottleneck of a decode step and choose between kernels, precision and parallelism to move it.

1. 06Anatomy of a GPUHBM, L2, SRAM, tensor cores — and why bandwidth rules inference.8 min
2. 08The RooflineArithmetic intensity: when you are memory-bound vs compute-bound.6 min
3. 09The KV CacheWhy it exists, how big it gets, and why it caps concurrency.7 min
4. 10FlashAttentionTiling + online softmax: attention without materialising N×N.8 min
5. 11Kernels, Fusion & CUDA GraphsLaunch overhead, kernel fusion and graph capture for tiny decode steps.7 min
6. 15Chunked PrefillToken budgets that stop long prompts from freezing everyone’s stream.6 min
7. 18QuantizationFP8, INT8, INT4, FP4: bit layouts, scales and what each speeds up.9 min
8. 17Speculative DecodingDraft cheaply, verify in one pass: draft models, EAGLE, Medusa, n-gram.7 min
9. 21Interconnects & CollectivesPCIe, NVLink, NVSwitch, InfiniBand and the all-reduce that rides on them.13 min
10. 22ParallelismTensor, pipeline, data and context parallelism and their communication.8 min
11. 23Mixture of ExpertsRouters, top-k experts, load imbalance and all-to-all.8 min
12. 24Disaggregated Prefill / DecodeSeparate pools for each phase, with KV transfer in between.7 min

You will be able to: Shape prompts for cache hits, constrain outputs, budget for reasoning and tools, and measure what users feel.

1. 01Text → TokensBPE merges, token IDs, chat templates and why token counts matter.8 min
2. 05Logits → Next TokenTemperature, top-k, top-p, min-p, penalties and greedy decoding.7 min
3. 07Prefill vs DecodeThe two phases of every request and why they behave so differently.7 min
4. 14Prefix CachingHashed blocks, radix trees and LRU eviction to skip repeated prefill.7 min
5. 19Structured OutputGrammar-constrained decoding by masking logits with a state machine.7 min
6. 31Reasoning Models & AgentsThousands of hidden tokens, multi-turn loops and tool calls.12 min
7. 32Multimodal ServingImages become tokens: encoder cost, token budgets and encoder caching.10 min
8. 33Embeddings & RerankersPrefill-only models: pooling, batching and serving them cheaply.11 min
9. 25Latency Metrics & GoodputTTFT, ITL, TPOT, E2E, p99, Little’s law and the goodput knee.8 min
10. 30Quality & SafetyProving an optimisation kept accuracy, and guarding inputs, outputs and weights.14 min