Token by Token
76 accelerators · 14 vendors · verified November 2026

Inference Hardware

Every GPU and accelerator worth knowing for LLM inference, from an RTX 4090 to a Vera Rubin rack, with the numbers that actually decide speed and cost. All figures are dense (no sparsity) and per chip. Click any row for its full spec sheet, tick up to four to compare, or jump to the calculator to see what your workload needs.

The five numbers that decide inference

A spec sheet has dozens of lines. For serving LLMs, five of them explain almost everything, and each one maps to a part of this guide.

1 · Memory capacityDecides which models fit at all and how much is left for the KV cache, which caps concurrency.GB2 · Memory bandwidthDecode re-reads every weight per step, so bandwidth sets the single-user speed limit: tok/s ≤ bandwidth ÷ weight bytes.TB/s3 · Dense low-precision FLOPsSets prefill speed (TTFT) and the ceiling for large-batch decode. FP8 and FP4 double it again, if your stack uses them.TFLOPS4 · Scale-up fabricNVLink, Infinity Fabric, ICI. Sets how cheaply a model splits across chips (tensor and expert parallelism) and how big that domain can be.GB/s × chips5 · Price and powerWhat you actually pay per token. A slower chip that costs a third as much can win on $/token while losing every benchmark.$/hr · W

Explorer

Filter by vendor or segment, sort any column, and click a row for the full spec sheet: compute per precision, memory, fabric, power, price, where to rent it, and the fine print. The last column shows the bandwidth-limited single-stream speed for the model you pick, on the fewest chips that hold it.

Capacity × bandwidth: the map of inference hardware

The two memory numbers explain most of inference. Up means bigger models and more KV cache; right means faster decode. The dashed lines show the time to read the entire memory once (capacity ÷ bandwidth). If you fill a chip with weights, that is the best-case time per decode step, so chips that sit high and to the left have lots of room but are slow to sweep it.

ChartMemory capacity vs memory bandwidth

Log–log scales. Faded markers are announced but not yet shipping. Unified-memory machines (Apple, DGX Spark, Strix Halo) sit high and left: plenty of room for a big model, but slow per token. SRAM-based chips (Groq, Cerebras) sit far right and low: enormous bandwidth but tiny capacity, so one model spans many chips. Click a point for its spec sheet.

The memory wall, chip by chip

The ridge point is peak FLOPs ÷ peak bandwidth: how many operations a kernel must do per byte it loads before compute becomes the limit. At each chip's fastest precision the ridge has climbed from about 300 (A100, INT8) to 600 (H100, FP8), 1,100 (B200, FP4) and 1,800 (Rubin, FP4), because every new number format doubles the FLOPs while HBM only grows steadily. Decode at batch B does about 2B FLOPs per weight, or B–4B FLOPs per byte depending on precision, so it sits in the shaded band, far below every modern ridge. That gap is why batching, quantization and speculative decoding exist.

ChartRidge point by year

Switch to BF16 only and the climb almost disappears: BF16 ridges have stayed between about 150 and 300 since 2020, because bandwidth kept pace with BF16 compute. The wall comes from the low-precision formats. Their extra FLOPs only help decode once the batch is big enough to use them; until then the chip is still waiting on memory. Faded markers are announced parts.

What should I run this on?

Describe the workload: model, precision, prompt and output length, how many requests run at once, and the slowest speed a user should see. For every priced accelerator, the calculator finds the smallest tensor-parallel group that holds all of those requests' KV cache and meets the speed target, then ranks the options by cost per million output tokens.

CalculatorHardware for your workload

The 2026 landscape

Five trends run through every vendor's roadmap, and each one shows up in the charts above.

Capacity per chip keeps climbing. 80 GB (H100) → 192 (MI300X) → 288 (B300, MI355X, Rubin) → 432 GB (MI455X). Whole 70B–400B models now fit on one chip or one node, and the leftover memory becomes KV cache.
FP4 is the new default for decode. Blackwell, MI355X, Trainium3, TPU 8 and Maia 200 all run 4-bit floats in hardware. The bytes per weight halve again, and so does the decode floor. See quantization.
The scale-up domain became a rack. NVLink went from 8 GPUs to 72 (NVL72), AMD's Helios also reaches 72, and TPU pods reach 9,216 chips. Tensor and expert parallelism can now span a rack without touching the network.
Prefill and decode are getting separate silicon. Google split TPU 8 into 8i (inference) and 8t (training). NVIDIA pairs Rubin with SRAM-based Groq 3 LPX racks for decode. Disaggregation moved from software into hardware.
Memory is the scarce input. An HBM and LPDDR shortage kept Hopper prices flat to rising through 2026 and pushed up desktop AI boxes (DGX Spark went from $3,999 to about $4,700, with reports near $7,000). Bandwidth and capacity remain what you are really paying for.

NVIDIA

  • Hopper (H100, H200, GH200) is still the most widely deployed inference GPU. H200's 141 GB and 4.8 TB/s make it the better decode part at the same compute.
  • Blackwell (B200, GB200) added native NVFP4, 8 TB/s of HBM3e and NVLink 5 at 1.8 TB/s per GPU. In GB200 NVL72, 72 GPUs share one NVLink domain.
  • Blackwell Ultra (B300, GB300) raises memory to 288 GB and dense FP4 to 13.5–15 PFLOPS, trading away most FP64 and INT8 throughput.
  • Rubin (Vera Rubin NVL72) began reaching clouds around August 2026 with HBM4. In September NVIDIA quietly cut the shipping specs from 22 to about 19.2 TB/s per GPU and NVLink 6 from 3.6 to 3.0 TB/s. The HGX Rubin NVL8 board carries 250 GB per GPU. Blackwell is still over 70% of 2026 shipments.
  • Rubin CPX, a GDDR7 prefill chip announced in 2025, was dropped after GTC 2026. In its place NVIDIA offers Groq 3 LPX racks (256 SRAM-based LPUs, from a roughly $20B licensing deal with Groq) that handle decode-heavy FFN and MoE work next to Vera Rubin. Rubin Ultra (4 dies, 1 TB HBM4E, Kyber racks) is planned for 2027, with reports of a slip to 2028.

AMD

  • MI300X (192 GB) was the first GPU to hold a 70B BF16 model alone. MI325X raised that to 256 GB. Both use the FNUZ variant of FP8, which is not bit-compatible with the OCP E4M3 that NVIDIA uses.
  • MI350X and MI355X (2025): 288 GB, 8 TB/s, OCP FP8, and MXFP4 and FP6 at the same rate, which suits 6-bit quantization unusually well.
  • MI455X and the Helios rack had their first revenue quarter in Q3 2026: 432 GB of HBM4 at 23.3 TB/s per GPU, with 72 GPUs joined by UALink over Ethernet. MI450X (288 GB) follows.
  • Software: ROCm 7 brought day-0 vLLM and SGLang containers, AITER kernels with MXFP4, and P/D disaggregation. Independent results put MI355X near B200 on many models, but parity still varies by model and engine version, so benchmark your own workload.

Google, AWS and the hyperscalers

  • TPU7x Ironwood (GA November 2025) is Google's first TPU designed around inference: 192 GB at 7.4 TB/s per chip, in pods of up to 9,216 chips. In April 2026 Google split the eighth generation into TPU 8i for inference (288 GB, 384 MB on-chip SRAM) and TPU 8t for training. TPUs serve through JAX and vLLM's TPU backend.
  • AWS Trainium2 (96 GB) and Trainium3 (GA December 2025: 144 GB, 4.9 TB/s, MXFP8 and MXFP4, 144-chip UltraServers; Anthropic is a launch customer) serve through the Neuron SDK and vLLM-Neuron. Inferentia2 remains a cheap option for small models.
  • Microsoft Maia 200 (January 2026: 216 GB, 10 PFLOPS FP4) and Meta MTIA 400 serve their owners' internal models and cannot be rented.

SRAM and dataflow chips

Groq, Cerebras and d-Matrix keep weights in on-chip SRAM instead of HBM. Per chip that means 80–150 TB/s (and 21 PB/s across a Cerebras wafer) but only hundreds of MB to a few GB of capacity, so one model spans hundreds of chips or several wafers. The payoff is single-stream speed: Cerebras serves 1T-parameter MoE models at roughly 1,000 tok/s, well past any GPU's bandwidth ceiling. Cerebras went public in May 2026 and powers OpenAI's Codex-Spark. d-Matrix Corsair reached volume production in June 2026. Most of these are sold as tokens through an API rather than as chips, so compare them on $/token and tok/s, not on the per-chip specs in the table.

Intel and Huawei

  • Intel Gaudi 3 still ships (OAM and PCIe) with modest adoption. Its dense BF16 figure was restated as 1,678 TFLOPS in 2025, down from the 1,835 quoted at launch. Falcon Shores was cancelled. Crescent Island, an inference GPU with 160 GB of LPDDR5X at 350 W, is sampling for a 2027 launch.
  • Huawei Ascend 910C powers the CloudMatrix 384 system (384 chips in one optical all-to-all domain). The 950PR and 950DT are Huawei's first chips with FP8 and MXFP4 and use in-house HBM, a response to export controls. Huawei publishes fewer figures than Western vendors, so treat those entries as estimates.

On your desk

  • RTX 5090 (32 GB, 1.8 TB/s) is the fastest single consumer card for local models. RTX PRO 6000 Blackwell (96 GB) runs a 70B model at FP8 on one card.
  • Apple M5 Ultra (Mac Studio, September 2026) reaches 512 GB of unified memory at about 1.2 TB/s, enough to run very large MoE models locally through MLX or llama.cpp.
  • DGX Spark (GB10, 128 GB at 273 GB/s), AMD Strix Halo (128 GB) and its 2026 refresh Gorgon Halo (192 GB, about 256 GB/s) trade speed for capacity: large models fit, but decode runs at a fraction of HBM speed. DGX Station GB300 puts 748 GB of coherent memory under a desk.

Reading a spec sheet without getting fooled

Sparse vs dense. NVIDIA and AMD headline FLOPs with 2:4 structured sparsity, which doubles the number. Almost no LLM serving stack uses it. Halve the headline; every number on this page is already dense.
Both directions summed. "900 GB/s NVLink" on an H100 is 450 GB/s each way. Collective maths (α–β model) needs the per-direction figure.
Marketed vs usable memory. B200 is marketed at 192 GB, but HGX systems expose 180 GB. Then the driver, CUDA context, activations and CUDA graphs take another 5–10%. Plan on about 90% of the label.
GB vs GiB. Spec sheets use decimal GB (10⁹). nvidia-smi and most engines report GiB (2³⁰), which is 7% larger, so "80 GB" shows up as about 79.6 GiB less reserved memory.
One name, several SKUs. B200 (1000 W, HGX) and the GB200 GPU (1200 W) have different FLOPs. H100 SXM, PCIe and NVL differ in bandwidth by almost 2×. Check the form factor before you compare.
FP4 is two formats. NVFP4 (Blackwell and later) uses 16-value blocks with an FP8 E4M3 scale. MXFP4 (OCP standard; AMD MI355X, Blackwell) uses 32-value blocks with a power-of-two scale. A checkpoint quantized for one is not automatically fast on the other. See quantization.
No FP8 tensor cores ≠ no FP8 model. A100 and A10 have no FP8 units. FP8 or INT4 weights still load (weight-only kernels such as Marlin upcast on the fly), so you get the memory and bandwidth savings but not the extra FLOPs.
GeForce accumulate rates. Consumer cards run FP16 and BF16 tensor maths with FP32 accumulation at half the rate of FP16 accumulation, and inference kernels need FP32 accumulation. Use the lower figure.
Unified memory is shared. On Apple silicon, DGX Spark and Strix Halo, "128 GB" is shared with the OS and CPU, and the bandwidth is a fraction of HBM's. Big models fit; tokens come slowly.
SRAM chips are a system, not a card. Groq and Cerebras hold weights in on-chip SRAM, so a 70B model spans hundreds of chips or a whole wafer. Compare them per rack or per token, not per chip.
Per chip, per board or per rack? Rack-scale numbers (NVL72, TPU pods) are often quoted for the whole system. This page normalizes to one chip (one GPU package, one TPU chip).
Cloud prices are a spread, not a number. The same H100 hour costs about $2 at a marketplace and $6+ at a hyperscaler on demand; commitments cut it by 30–60%. The ranges here are on-demand, late 2026.

Method & sources

Specs come from vendor datasheets, product pages and architecture whitepapers, cross-checked against Hot Chips papers and independent analyses (Chips and Cheese, ServeTheHome, SemiAnalysis, TechPowerUp). Each spec sheet lists its own sources. Where a vendor publishes only sparse figures they are halved; where a value could not be verified it is left blank rather than guessed. Prices are typical on-demand ranges from public price lists and GPU price indexes, gathered in November 2026; treat them as indicative. Derived figures (ridge point, decode ceiling, fit) use the same first-order model as the GPU chapter and the Lab: 75% of peak bandwidth and 90% of memory usable.

Found a wrong number? The data lives in one file, src/data/accelerators.ts, and every calculator on the site reads from it.