Interconnects & Collectives

Once a model spans more than one GPU, some data has to leave the GPU it was computed on. Every chapter in this part is built on that fact:

• Tensor parallelism sums partial results with an all-reduce, twice per layer, for every token.
• Expert parallelism moves tokens to their experts and back with an all-to-all, twice per MoE layer.
• Pipeline parallelism and disaggregated prefill/decode send activations or KV caches point-to-point.

Whether these cost microseconds or dominate the step depends on which wire the bytes cross. This chapter is about those wires, the collective operations that run over them, and the software (mostly NCCL) that picks the path.

The link hierarchy

A GPU sits at the centre of a set of links that differ by orders of magnitude. From fastest to slowest:

Two traps in that table. NVLink is marketed as the sum of both directions, while PCIe and NICs are usually quoted per direction. Network links are quoted in bits: 400 Gb/s is 50 GB/s. Normalise before comparing.

An NVSwitch turns NVLink from point-to-point cables into a crossbar: in an 8-GPU HGX node every GPU reaches every other GPU at full NVLink bandwidth at the same time. The set of GPUs that can talk over NVLink is the NVLink domain (the scale-up domain). Classically that is one 8-GPU server. GB200 NVL72 extends it to a whole rack: 72 GPUs and 36 Grace CPUs in one NVLink domain with roughly 130 TB/s of aggregate bandwidth. Everything outside the domain is scale-out: NICs and a network fabric.

Latency vs bandwidth: the α–β model

The simplest useful model of any transfer is

where $S$ is the message size, $\beta$ the link bandwidth, and $\alpha$ a fixed latency per message: kernel launch, synchronisation between GPUs, protocol handshakes and wire delay. α is a few microseconds on NVLink and somewhat more over RDMA; over TCP it is tens of microseconds. Messages much smaller than $\alpha \cdot \beta$ are latency-bound: doubling the bandwidth does almost nothing. Messages much larger are bandwidth-bound.

A collective is a sequence of such transfers. A ring all-reduce over $N$ GPUs takes $2(N-1)$ steps, each moving $S/N$:

The bandwidth term is nearly independent of $N$: each GPU sends just under $2S$ however many GPUs there are. That is why ring is the standard algorithm for large messages. The latency term, though, grows linearly with $N$. A tree algorithm reaches every GPU in about $2\log_2 N$ steps, so it wins for small messages and large $N$, at the cost of slightly less bandwidth efficiency.

This explains a fact from the parallelism chapter in one line: a 70B model at TP = 8 runs 160 all-reduces per decode step, each about 14 ring steps. With α around 1.5 µs that is already ~3 ms of pure latency per token, before a byte of bandwidth is counted. Over a network with 5–30 µs per step, it would be the whole step budget.

The collectives inference uses

A collective is a communication pattern that every GPU in a group joins at once. Five of them cover nearly everything an inference engine does:

The tensor-parallel diagram animates a ring all-reduce chunk by chunk. All-to-all is the awkward one: message sizes depend on the router’s choices, so they change every step and are rarely balanced, and every GPU talks to every other GPU, which stresses the whole fabric rather than neighbour links.

Topology: what is wired to what

“Eight GPUs in a box” can mean very different things. In an HGX-style node every GPU pair is joined through NVSwitch. In a PCIe server, GPUs hang off PCIe switches under two CPU sockets, and a GPU-to-GPU copy may cross switches, a CPU host bridge or even the link between sockets. nvidia-smi topo -m prints a matrix that labels each pair:

What to take from the matrix:

• TP groups must be NV# pairs, or at worst PIX. A TP = 4 group spread across SYS pairs on a PCIe box is the classic way to make a model slower on more GPUs.
• Each GPU should use its nearest NIC (PIX), so RDMA traffic goes NIC ↔ GPU memory through one PCIe switch. Large nodes typically have one 400 Gb/s NIC per GPU, cabled so that GPU i of every node shares a network “rail”.
• NUMA matters for the CPU side too. The matrix also prints each GPU’s CPU and NUMA affinity. Engine processes, tokenisers and pinned host buffers for a GPU should live on that GPU’s socket, otherwise every host ↔ device copy crosses the socket link.

NCCL: the software that picks the path

Almost every engine (vLLM, SGLang, TensorRT-LLM) uses NCCL (on AMD, RCCL) for collectives. At start-up NCCL discovers the topology, then picks a transport for each pair of ranks and an algorithm and protocol for each collective:

• P2P: direct GPU-to-GPU copies over NVLink or PCIe.
• SHM: staging through host shared memory, used when P2P is unavailable or slow (often across SYS).
• NET: to other nodes, through the IB plugin (InfiniBand or RoCE, with RDMA) or plain Socket (TCP).
• Algorithms such as Ring, Tree and NVLS (in-switch reduction on NVSwitch systems), and protocols LL, LL128 and Simple, which trade latency against bandwidth.

When performance is mysteriously bad, the first move is NCCL_DEBUG=INFO. The log shows the detected topology and, for each channel, which transport each connection uses: a line ending in via NET/Socket where you expected via NET/IB, or via SHM where you expected P2P, explains a 5–10× slowdown on its own. Useful knobs exist (NCCL_SOCKET_IFNAME, NCCL_IB_HCA, NCCL_P2P_LEVEL, NCCL_NET_GDR_LEVEL, NCCL_ALGO, NCCL_PROTO), but they are for diagnosis first and tuning second.

Two hardware paths decide whether inter-node traffic touches the CPU at all:

• GPUDirect RDMA: the NIC reads and writes GPU memory directly, with no bounce through host RAM. It needs the right kernel support (the nvidia-peermem module or DMA-BUF) and a NIC close to the GPU. KV transfer libraries such as NIXL and Mooncake rely on it.
• GPUDirect Storage (GDS): NVMe ↔ GPU memory directly through the cuFile API, which speeds up weight loading when the storage stack supports it.

Placing parallelism on the hierarchy

Match each parallelism strategy’s traffic to the link that can carry it:

The NVLink domain is the boundary that matters. Inside an 8-GPU node, TP up to 8. Across nodes, add PP or DP, or with an NVL72 rack, keep TP and wide EP inside one rack-scale domain (on Kubernetes that domain is requested explicitly; see GPUs on Kubernetes).

The final lever is overlap: hide communication behind computation that does not depend on it.

• Layer-by-layer KV transfer: send layer k’s KV while layer k + 1 is still computing (disaggregation).
• Two micro-batches ping-ponging: one runs its all-to-all while the other computes its experts, as large MoE deployments do.
• Pipeline stages overlap naturally once several micro-batches are in flight.
• Fused GEMM + collective kernels start sending the first tiles of a matmul while later tiles are computing.

Overlap only helps when there is independent work and spare link and SM capacity. In decode at small batch there is little of either, which is why the raw α of the link still matters.