Anatomy of a GPU
Thousands of math units, a ladder of memories from tiny-and-fast to huge-and-slow, and why LLM inference is mostly a fight over memory bandwidth.
A GPU is a machine for doing the same multiply-add on huge arrays at once. For LLM serving, two of its numbers matter more than all the others: how many FLOPs it can do per second, and how many bytes it can move out of memory per second. Almost every optimisation in the rest of this guide is a way of balancing those two.
The chip, top to bottom
A modern data-centre GPU contains:
- Streaming Multiprocessors (SMs): 100+ independent cores. Each SM holds tensor cores (hardware for small matrix multiplies), a large register file, and a block of fast on-chip shared memory (SMEM, also used as L1 cache).
- L2 cache: tens of MB, shared by every SM.
- HBM (High Bandwidth Memory): stacks of DRAM packaged next to the die. This is the “80 GB” on the spec sheet, and it is where the weights and the KV cache live.
- Interconnects: NVLink (or AMD Infinity Fabric) to other GPUs in the same server, and PCIe to the host CPU, its RAM and storage.
Pick a GPU and click a level to highlight where it sits on the chip. Each level going down is roughly 10–1000× bigger and several times slower than the one above it. Values marked ≈ are rough estimates derived from SM counts; HBM and link figures are spec-sheet peaks.
The famous imbalance
Take an H100 SXM. It can do about 989 trillion BF16 FLOPs per second, but it can read only about 3.35 trillion bytes per second from HBM. Divide one by the other:
To keep the tensor cores busy, every byte fetched from HBM has to be used in roughly 300 floating-point operations. If a workload does less math per byte than that, the tensor cores sit idle waiting for data. That workload is memory-bound. This ratio becomes the “ridge point” in the roofline chapter.
A decode step, seen from the chip
To generate one token, the model has to multiply the current hidden state by every weight matrix in every layer. For an 8B-parameter model in BF16 that is 16 GB of weights crossing the HBM bus for a single token. Each weight is used for only 2 FLOPs (one multiply, one add) per token in the batch.
The top lane shows HBM moving weights; the bottom lane shows tensor cores doing math. At batch 1 the tensor cores are idle more than 99% of the time. Raise the batch: every weight read is now shared by many sequences, so throughput climbs almost for free until compute finally catches up.
The animation shows the central fact of LLM inference: single-stream decode speed is set by memory bandwidth, not by FLOPs. A GPU with twice the TFLOPS but the same bandwidth would produce a single user’s tokens no faster.
Your speed limit, calculated
Because decode must read every weight once per step, there is a hard lower bound on step time:
Real kernels reach about 70–90% of peak bandwidth; this fraction is called MBU, memory-bandwidth utilisation. Compute-bound work typically reaches 40–70% of peak FLOPs, called MFU.
In the shaded memory-bound region, doubling the batch nearly doubles throughput, because the weight read is shared. Past the knee the step becomes compute-bound and throughput flattens. To keep the model simple, this ignores KV-cache reads, which also grow with batch and context; see the KV cache. For MoE models only the experts actually routed to are read, so small batches are cheaper than the total parameter count suggests.
Things to try in the calculator:
- Llama 3.1 70B on one H100: the weights do not fit. Switch to 2 or 4 GPUs. With tensor parallelism, each GPU reads only its own shard, so bandwidth adds up across GPUs.
- INT4 weights: the single-stream ceiling roughly quadruples, because there are 4× fewer bytes to read. This is why weight-only quantization is so popular for low-batch serving.
- H100 vs H200: identical FLOPs but 43% more bandwidth. Decode gets faster; compute-bound prefill does not.
What this means for serving
| Fact about the hardware | Consequence for serving |
|---|---|
| FLOPs ≫ bandwidth (≈300 FLOP/byte) | Small-batch decode wastes almost all compute |
| Weights are read once per step regardless of batch | Batching is the main throughput lever |
| HBM capacity is finite | Weights + KV cache must fit, and KV caps concurrency |
| SMEM is tiny but fast | Kernels like FlashAttention tile work to stay on-chip |
| Kernel launches cost microseconds | Tiny decode steps need fusion and CUDA graphs |
| NVLink ≫ PCIe | Tensor parallelism stays inside one NVLink domain |
coreAn H100 has ~1,000 TFLOPS of BF16 compute. Why can't it generate 1,000 tokens/s for one user on an 8B model?
Each token needs the whole 16 GB of BF16 weights read from HBM. At 3.35 TB/s that takes at least ≈4.8 ms, which caps a single stream at about 200 tok/s, and nearer 160 at realistic bandwidth utilisation. The math itself takes microseconds. Decode is bandwidth-bound, not compute-bound.
coreWhat is the difference between HBM, L2 and shared memory?
HBM is the large off-chip DRAM (tens to hundreds of GB) that holds weights and KV cache, at a few TB/s. L2 is a tens-of-MB on-chip cache shared by all SMs. Shared memory is a small (around 100–228 KB) per-SM scratchpad that kernels manage explicitly, with tens of TB/s of aggregate bandwidth. Fast kernels move data from HBM into SMEM once and reuse it there.
midDefine MBU and MFU. Which one should you watch for decode, and which for prefill?
MBU = achieved bytes/s ÷ peak bandwidth. MFU = achieved FLOPs/s ÷ peak FLOPs. Decode is memory-bound, so judge it by MBU (good kernels reach about 80%). Prefill and large-batch work are compute-bound, so judge them by MFU. A decode MFU of 1% is expected, not a bug.
midH100 → H200 upgrade: which workloads get faster, and roughly by how much?
The H200 has the same compute but about 1.43× the bandwidth (4.8 vs 3.35 TB/s) and 141 GB of memory. Memory-bound decode gets up to about 1.4× faster per step. The extra capacity also means more KV cache, so larger batches fit. Compute-bound prefill barely changes.
seniorGPU utilisation shows 100% in nvidia-smi but throughput is poor. How can both be true?
That “utilisation” figure only means some kernel was running during each sample window. It says nothing about how many SMs were busy or how close they were to peak FLOPs or bandwidth. Look at SM activity / occupancy and DRAM bandwidth utilisation (for example via DCGM profiling metrics). Also check the achieved batch size, preemptions, and time spent in communication.
seniorWhy does a MoE model with 37B active parameters decode faster than a dense 70B model, but need far more memory?
Per token, decode reads only the active experts’ weights: about 37B parameters versus 70B, so fewer bytes per step. But all experts must stay resident in memory (671B parameters for DeepSeek-V3), so capacity, not bandwidth, becomes the constraint. As the batch grows, more distinct experts are touched each step, and the per-step byte count climbs back toward the full model.