Cheat Sheet · Token by Token

The model

Tokens, attention, one decoder layer, and the FLOP count every estimate starts from.

Scaled dot-product attention03 · Attention

\text{Attention}(Q,K,V) = \text{softmax}\left(\frac{QK^\top}{\sqrt{d_k}} + M\right) V

M = causal mask: 0 where a token may look, −∞ where it may not; d_k = key dimension

One decoder layer (pre-norm residual, twice)04 · Transformer layer

h = x + \text{Attention}(\text{RMSNorm}(x)), \qquad x' = h + \text{MLP}(\text{RMSNorm}(h))

x = residual stream of shape [n, d_model]; the layer repeats L times, then a final norm and the LM head

FLOPs per token04 · Transformer layer

\text{FLOPs per token} \approx 2 \times N_\text{active} + 4 \, L \, n \, d_\text{attn}

N_active = active parameters; L = layers; n = cached tokens attended over; the 2 is one multiply plus one add per weight

Example8B model: 2 × 8e9 = 16 GFLOPs per token, against about 16 GB of weights read: roughly 1 FLOP per byte

Softmax with temperature05 · Sampling

p_i = \frac{e^{z_i / T}}{\sum_j e^{z_j / T}}

z = logits; T → 0 is greedy, T = 1 is the model’s own distribution, T > 1 is flatter

RoPE frequencies02 · Embeddings & RoPE

\theta_i = \text{base}^{-2i/d_\text{head}}, \qquad i = 0, 1, \dots, \frac{d_\text{head}}{2} - 1

pair i of a token at position m is rotated by m × θ_i; scores then depend only on the distance between positions

Exampled_head = 128 gives 64 pairs; Llama 3 uses base 500,000

Vocabulary size	about 128K (Llama 3), about 150K (Qwen3)	One logit per vocabulary entry is produced for every generated token.	01
English text per token	about 4 characters, or 0.75 words	For a modern ~128K vocabulary; code, JSON, other languages and numbers cost more.	01
Llama 3.1 layer count	32 layers (8B), 80 layers (70B)	The same decoder layer applied L times, each with its own weights.	04
Llama 3.1 8B attention heads	32 query heads, d_head = 128	Llama 3 uses 8 KV heads for 32 or 64 query heads (GQA).	03
Llama 3.1 8B embedding table	128,256 × 4,096 ≈ 525M parameters, about 1 GB in BF16	Around 6.5% of the model; a lookup (gather), not a matmul.	02
Where the parameters are (dense LLM)	MLPs: about two thirds	Attention projections take most of the rest; embeddings plus LM head a few percent.	04

Memory: weights and KV cache

Weights are paid once; KV cache is paid per request, per token, and caps concurrency.

KV cache per token09 · KV cache

\text{KV bytes/token} = 2 \times n_\text{layers} \times n_\text{kv heads} \times d_\text{head} \times \text{bytes per element}

the 2 is K and V; n_kv heads = KV heads (not query heads); bytes per element = 2 in BF16

ExampleLlama 3.1 8B in BF16: 2 × 32 × 8 × 128 × 2 = 131,072 bytes = 128 KiB per token

Weight memory29 · Capacity & cost

\text{weight bytes} = N_\text{params} \times \text{bytes per weight}

Example14B model in BF16: 14B × 2 B = 28 GB

Max concurrent sequences (memory ceiling)29 · Capacity & cost

\text{max sequences} \approx \frac{\text{memory budget} - \text{weights} - \text{runtime overhead}}{\text{KV bytes/token} \times \text{context tokens}}

memory budget = gpu_memory_utilization × VRAM; everything left after weights and overhead becomes KV blocks

Example80 × 0.9 = 72 GB; minus 28 GB weights and 3–4 GB overhead leaves ~40 GB; at 160 KiB/token ≈ 250k tokens ≈ 60 conversations at 4k

Llama 3.1 8B KV cache (BF16)	128 KiB per token	One 8K-token conversation needs 1 GiB.	09
Llama 3.1 70B KV cache (BF16)	320 KiB per token	An 8K-token prompt produces 2.5 GiB.	24
8B model weights in BF16	16 GB	All of it crosses the HBM bus for every decode step.	06
Llama 3.1 70B weights in BF16	141 GB	An H100 has 80 GB, so it needs more than one GPU.	22
GQA vs MHA cache size	4–16× smaller	Groups of query heads share one K/V head; tiny quality cost, now standard.	09
MLA cache size	~4× smaller than GQA-8, ~50× smaller than MHA	Caches one compressed latent instead of K and V.	09
gpu-memory-utilization	0.90–0.95	As high as is stable, so KV gets everything left over.	28

Speed: bandwidth, FLOPs and the roofline

Decode is bound by bytes read per step, prefill by FLOPs; the ridge point says which one applies.

Decode step time bound06 · GPU & memory

t_\text{step} \ge \frac{\text{bytes of weights read}}{\text{memory bandwidth}} \qquad \Rightarrow \qquad \text{tok/s per sequence} \le \frac{\text{bandwidth}}{\text{weight bytes}}

decode must read every weight once per step, whatever the batch size

Example8B in BF16 on H100: 16 GB ÷ 3.35 TB/s ≈ 4.8 ms, so about 200 tok/s for one stream, nearer 160 at realistic bandwidth utilisation

Decode step time with KV cache12 · Batching

t_\text{step} \approx \frac{\text{weight bytes} + \sum_\text{seqs} \text{KV bytes}}{\text{memory bandwidth}}

the weight term is shared by the whole batch; only the KV term grows with batch size and context

Arithmetic intensity08 · Roofline

I = \frac{\text{FLOPs performed}}{\text{bytes moved to/from HBM}}

I in FLOP/byte

Roofline: attainable throughput08 · Roofline

\text{attainable FLOP/s} = \min(P, \; B \times I)

P = peak compute (FLOP/s); B = memory bandwidth (byte/s); I = arithmetic intensity

Ridge point08 · Roofline

I^{\star} = \frac{P}{B}

below the ridge an operation is memory-bound, above it compute-bound

ExampleH100 in BF16: 989 TFLOPS ÷ 3.35 TB/s ≈ 295 FLOPs per byte

Decode matmul intensity = batch size08 · Roofline

I_\text{decode} \approx \frac{2 b d^2}{2 d^2} = b \quad \text{(BF16)}, \qquad I_\text{decode} \approx 2b \quad \text{(FP8)}

b = sequences decoding together (or prompt tokens, for prefill); d = hidden size

ExampleBatch 32 in BF16 ≈ 32 FLOP/byte against an H100 ridge of about 295: memory-bound, tensor cores at most about 11% busy

Decode attention intensity08 · Roofline

I_\text{attn} \approx \frac{2g}{\text{bytes per element}}

g = query heads sharing each K/V head; batching does not change it because every sequence brings its own cache

Exampleg = 4 in BF16 gives I ≈ 4

Prefill cost07 · Prefill vs decode

\text{prefill FLOPs} \approx 2 \times N_\text{params} \times n_\text{prompt}

divide by achieved FLOP/s for time; a quadratic attention term is added for long prompts

Example4,000-token prompt, 8B model, one H100: 2 × 8×10⁹ × 4,000 = 6.4×10¹³ FLOPs; at about 55% of 989 TFLOPS that is roughly 120 ms

MBU and MFU06 · GPU & memory

\text{MBU} = \frac{\text{achieved bytes/s}}{\text{peak bandwidth}}, \qquad \text{MFU} = \frac{\text{achieved FLOPs/s}}{\text{peak FLOPs}}

judge decode by MBU and prefill or large-batch work by MFU

ExampleGood decode kernels reach about 80% MBU; a decode MFU of 1% is expected, not a bug

H100 SXM BF16 compute	about 989 TFLOPS	The compute roof of the roofline.	06
H100 HBM bandwidth	3.35 TB/s	Sets single-stream decode speed.	06
H100 ridge point (BF16)	≈ 295 FLOPs per byte	Roughly 300 sequences must decode together before matmuls become compute-bound.	08
Batch-1 decode intensity	about one FLOP per byte	An H100 needs around 300 FLOPs per byte to keep its tensor cores busy.	04
H200 vs H100	4.8 vs 3.35 TB/s (about 1.43×), 141 GB, same compute	Memory-bound decode gets faster; compute-bound prefill barely changes.	06
Ridge at each chip’s best precision	A100 BF16 ~150, H100 FP8 ~590, B200 FP4 ~1,100, Rubin FP4 ~1,800 FLOPs per byte	The batch needed to saturate the tensor cores keeps growing.	08
Achieved bandwidth (MBU)	70–90% of peak	What real memory-bound kernels reach.	06
FlashAttention vs naive attention	2–4× faster, O(N) extra memory	Exact, not an approximation: same FLOPs, far fewer bytes moved.	10
CUDA graphs on small-batch decode	commonly 1.5–3×	Eager frameworks add 10–30 µs of Python and dispatcher overhead per op.	11

The engine: batching, paging, caching, scheduling

How one engine keeps many sequences in every step without wasting KV memory or stalling streams.

Useful slot-time in a locked batch12 · Batching

\text{useful} = \frac{\text{mean}(L_1, \dots, L_B)}{\max(L_1, \dots, L_B)}

L_i = output length of request i; depends on the spread of lengths, not their scale

PagedAttention block lookup13 · PagedAttention

\text{physical block} = \text{block table}[\lfloor t / B \rfloor], \qquad \text{slot} = t \bmod B

t = token position; B = block size in tokens; waste is at most B − 1 slots per sequence

Prefix-cache block hash (chained)14 · Prefix caching

h_i = \text{hash}(h_{i-1}, \; \text{tokens in block } i, \; \text{extra keys})

extra keys = anything else that changes K/V, such as a LoRA adapter id or multimodal inputs

Chunked prefill token budget15 · Chunked prefill

\text{chunk} = \text{budget} - n_\text{decoding}, \qquad \text{iterations to prefill} = \left\lceil \frac{\text{prompt length}}{\text{chunk}} \right\rceil

budget = max_num_batched_tokens; n_decoding = running decode sequences, which are served first

KV offload: reload vs recompute16 · Scheduler

\frac{\text{tokens} \times \text{KV bytes/token}}{\text{tier bandwidth}} \quad \text{vs.} \quad \frac{2 P N + 2 L N^2 H d}{\text{GPU FLOP/s} \times \text{MFU}}

left = reload time from a slower tier; right = recompute the prefill; reload grows linearly, recompute faster than linearly

Default KV block size	16 tokens	Keeps tail waste under a few percent; some backends prefer 32, 64 or larger.	13
KV memory holding real tokens, contiguous allocation	20–40%	Measured on early systems that reserved max-length buffers.	13
KV utilization with paging	above 95%	Waste is limited to the last, partially filled block of each sequence.	13
Prefix caching default	On (vLLM automatic prefix caching and SGLang RadixAttention)	Only full blocks are cached; matching is exact and positional.	14
Chunked prefill budget vs stream stall	no chunking: 400+ ms freeze; 2048: ~150 ms hiccups; 512: gaps under ~45 ms	Smaller budget: smoother ITL, later first token for the long prompt.	15
Token budget for interactive chat	low thousands of tokens	Tune against p99 ITL and p95 TTFT on your own traffic.	15
kv_cache_dtype FP8	roughly 2× KV capacity	Possible accuracy cost.	16

Faster and cheaper

Speculative decoding, quantization, constrained output and LoRA, with the arithmetic behind each.

Speculative decoding: acceptance rule17 · Speculative decoding

P(\text{accept } x) = \min\left(1, \frac{p(x)}{q(x)}\right)

p = target distribution; q = draft distribution; on rejection, resample from the residual max(0, p − q), renormalised

Expected tokens per target pass17 · Speculative decoding

\mathbb{E}[\text{tokens}] = \frac{1 - \alpha^{k+1}}{1 - \alpha}

α = probability a draft token is accepted (overlap of p and q); k = draft length

Exampleα = 0.8 and k = 4 gives about 3.4 tokens per pass

Speculative decoding speedup (batch 1)17 · Speculative decoding

\text{speedup} \approx \frac{\mathbb{E}[\text{tokens}]}{1 + k \, c}

c = cost of one draft step as a fraction of a target step

Exampleα = 0.7, k = 4: (1 − 0.7⁵) / (1 − 0.7) ≈ 2.77 tokens; drafter at 5% of a target step: 2.77 / (1 + 4 × 0.05) ≈ 2.3×

Integer quantization with a scale18 · Quantization

q = \text{round}\left(\frac{w}{s}\right), \qquad s = \frac{\max |w|}{2^{b-1} - 1}, \qquad \hat{w} = q \cdot s

w = weights sharing one scale; b = bits; s = scale set by the absmax, so one outlier stretches the grid

Constrained decoding: mask the logits19 · Structured output

\tilde{z}_i = z_i \ \text{if output + token}_i \text{ is a valid prefix, else } -\infty, \qquad p = \text{softmax}(\tilde{z} / T)

guarantees every prefix is valid, not that the output is finished or correct

LoRA20 · Multi-LoRA

W' = W + \frac{\alpha}{r} B A, \qquad y = x W^\top + \frac{\alpha}{r} (x A^\top) B^\top

W frozen, d_out × d_in; A is r × d_in; B is d_out × r; r much smaller than d; W′ is never formed at inference

LoRA adapter size20 · Multi-LoRA

\frac{\text{adapter params}}{\text{matrix params}} = \frac{2 r d}{d^2}

for a d × d matrix: r·2d parameters instead of d²

Exampler = 16 and d = 4096: about 0.8% of the matrix

Draft length k in most deployments	between 2 and 6	Longer drafts help only while α is high.	17
Acceptance rate α by workload	above 0.8 for code completion and structured extraction; near 0.5 for creative chat	Measure acceptance rate and mean accepted length in production.	17
BF16 bit layout (S/E/M)	1 / 8 / 7	Keeps FP32’s 8-bit exponent, so the same range.	18
FP8 E4M3 bit layout and max	1 / 4 / 3, max 448	Inference weights and activations.	18
Scale granularity for 4-bit weights	one scale per group of 32–128 weights	An FP16 scale per 128 weights costs 0.125 bits per weight.	18
Low-precision tensor-core rate	2× (8-bit) or 4× (4-bit) the BF16 rate	Only when both weights and activations are low precision (W8A8, W4A4).	18
FP8 KV cache	halves cache memory per token	Roughly twice the concurrent sequences or context; risk is long-context accuracy.	18
Typical LoRA rank	r = 8–64	A whole-model adapter is often tens of MB.	20
Multi-LoRA decode overhead	single-digit-percent slower than the plain base model	A cold adapter miss adds hundreds of milliseconds to TTFT.	20

Scaling out: links, collectives, TP / PP / EP

Which wire the bytes cross decides whether communication costs microseconds or the whole step.

α–β transfer model21 · Interconnects

T(S) = \alpha + \frac{S}{\beta}

S = message size; β = link bandwidth; α = fixed latency per message; messages much smaller than α·β are latency-bound

Ring all-reduce cost21 · Interconnects

T_\text{ring} = 2(N-1) \, \alpha + \frac{2(N-1)}{N} \cdot \frac{S}{\beta}

N = GPUs; 2(N−1) steps each moving S/N; the latency term grows linearly with N, the bandwidth term barely changes

Example70B at TP = 8: 160 all-reduces per decode step, each about 14 ring steps; with α around 1.5 µs that is ~3 ms of pure latency per token

Bus bandwidth (nccl-tests)21 · Interconnects

\text{busbw} = \text{algbw} \times \frac{2(N-1)}{N}

algbw = S ÷ time; the factor is the all-reduce traffic per GPU, so busbw compares directly with link bandwidth

All-reduces per TP decode step22 · TP / PP / DP

\text{all-reduces per decode step} = 2 \times n_\text{layers}

one after attention and one after the MLP, in every layer, for every generated token

Example80-layer 70B model: 160 per step; each message is batch × hidden × 2 bytes, about 0.5 MB for a batch of 32

TP decode step time (first order)22 · TP / PP / DP

t_\text{step} \approx \frac{\text{weights} + \text{KV read per GPU}}{\text{HBM bandwidth}} + 2 \times n_\text{layers} \times t_\text{all-reduce}

with TP = n each GPU reads only 1/n of the weights, and the GPUs read in parallel

Pipeline bubble22 · TP / PP / DP

\text{idle fraction} = \frac{P - 1}{M + P - 1}

P = pipeline stages; M = micro-batches in flight

MoE experts touched per step23 · MoE & expert parallel

\text{fraction of experts touched} \approx 1 - \left(1 - \frac{k}{E}\right)^{\text{batch}}

k = experts per token; E = experts per layer; assumes uniform routing

P/D disaggregation: KV transfer time24 · P/D disaggregation

t_\text{transfer} \approx \frac{n_\text{prompt} \times \text{KV bytes/token}}{\text{link bandwidth}}

Example70B, 8K prompt: 320 KiB × 8,192 tokens ≈ 2.5 GiB; at ~50 GB/s (400 Gb/s RDMA) that is ~54 ms, against 150–300 ms of prefill

NVLink 5 (B200)	1.8 TB/s (both directions)	72 GPUs in one domain on GB200 NVL72.	21
NVLink 4 (H100 / H200)	900 GB/s (both directions)	NVLink is quoted as the sum of both directions: 450 GB/s each way.	21
PCIe Gen5 x16	~64 GB/s per direction	GPU to CPU, NIC and NVMe; Gen4 x16 is ~32 GB/s.	21
InfiniBand XDR / NDR	800 / 400 Gb/s per NIC = ~100 / ~50 GB/s	Network links are quoted in bits; normalise before comparing.	21
H100 link ratios	HBM ≈ 7× NVLink; NVLink ≈ 9× a 400 Gb/s NIC and ≈ 7× PCIe Gen5	Keep frequent communication off the slow end of the ladder.	21
TP limit for Llama 3.1	8 KV heads, so TP = 16 replicates KV heads	Standard layout: TP inside a node, PP or DP across nodes.	22
DeepSeek-V3 parameters	671B total, about 37B used per token	Memory scales with total parameters, compute with active parameters.	23
Weight memory in BF16	Dense 70B ~141 GB; MoE 30B-A3B ~61 GB; MoE 671B-A37B ~1.3 TB (≈ 671 GB in FP8)		23
Prefill and decode pool sizing	below about 70–80% utilisation at peak	Decode concurrency = rate × output length × ITL.	24

Production: latency, goodput, capacity, cost

From SLOs and traffic to replicas and cost per million tokens.

TPOT from E2E and TTFT25 · Metrics & goodput

\text{TPOT} = \frac{\text{E2E} - \text{TTFT}}{N_\text{out} - 1}

N_out = output tokens; TPOT is a per-request average, ITL is measured per gap

End-to-end latency25 · Metrics & goodput

\text{E2E} \approx \text{TTFT} + (N_\text{out} - 1) \cdot \text{TPOT}

TTFT ≈ queueing + prefill; TPOT ≈ one decode step

Goodput25 · Metrics & goodput

\text{goodput} = \frac{\text{requests with TTFT} \le \text{SLO}_\text{TTFT} \text{ and TPOT} \le \text{SLO}_\text{TPOT}}{\text{time window}}

the rate of requests that finished within every SLO; capacity is the knee of this curve, not peak throughput

Little's law25 · Metrics & goodput

L = \lambda \times W

L = average requests in flight; λ = arrival rate; W = average time each spends inside

Example4 req/s with 10 s average E2E means about 40 requests in flight; at 24 per replica within SLO, 2 replicas before headroom

Requests per second per replica29 · Capacity & cost

\text{req/s per replica} = \frac{1}{\dfrac{N_\text{out} \, t_\text{step}(B)}{B} + t_\text{prefill}}

B = batch; t_step(B) ≈ (weight bytes + B × average context × KV/token) ÷ HBM bandwidth; B is capped by memory and the TPOT SLO

Replica count29 · Capacity & cost

\text{replicas} = \left\lceil \frac{\lambda}{\text{req/s per replica} \times \text{target utilisation}} \right\rceil

λ = peak request rate; target utilisation is the headroom, e.g. 70% of the knee

Cost per million output tokens29 · Capacity & cost

\text{cost per M tokens} = \frac{\text{GPUs} \times \text{price per GPU-hour}}{\text{output tokens per hour} / 10^6}

Example2 GPUs at $2.50/h = $5/h; 1,500 output tok/s is 5.4M tokens/h, so about $0.93/M fully loaded, about $2.30/M at 40% utilisation

Percentiles to report	p50, p95, p99	Prompt lengths vary by 100×, output lengths by 50×; averages hide the tail.	25
Fleet percentile from replicas	histogram_quantile(0.95, sum by (le) (rate(...bucket[5m])))	Merge histogram buckets first; averaging per-replica p95s is wrong.	25
Daily-cycle cost multiplier	30–40% average utilisation makes effective cost 2.5–3× the peak-hour figure	Utilisation is the hidden multiplier on cost per token.	29
Requests per sweep point for a p99	at least a few hundred, ideally 1,000+	With 100 requests the p99 is just the slowest request.	26
Typical ISL : OSL	chat ~3:1; RAG 10–20:1; summarisation 15–30:1; code completion 30–50:1; reasoning 1:5–10; agents 50+:1		26
Routing to a replica that holds the prefix	can cut prefill work by 10×	One request can occupy a replica 1,000× longer than another.	27
Autoscaling signals	scale on KV-cache usage or queue depth; not CPU or headline GPU utilisation	vllm:kv_cache_usage_perc rises before vllm:num_requests_waiting does.	27
Eval size to resolve a 1-point change	on the order of 1,000+ paired examples per task	Gate per task, never on the average.	30

Running on Kubernetes

GPU sharing, getting weights into HBM, and the signals worth knowing by heart.

Weight load time from a shared source36 · Weights & loading

t_\text{load} \approx \frac{\text{checkpoint bytes}}{\text{bandwidth per replica}}, \qquad \text{bandwidth per replica} = \frac{\text{shared source bandwidth}}{\text{replicas starting together}}

shared sources (hub egress, NAT gateway, NFS filer) split their bandwidth; object stores and local disks barely change

Example140 GB checkpoint: about 8 minutes from the internet, about 25 seconds from local NVMe; 20 replicas on a shared source are each roughly 20× slower

Startup probe budget28 · Serving stack

\text{failureThreshold} \times \text{periodSeconds} > \text{worst-case download} + \text{load} + \text{compile time}

readiness should pass only after warm-up; liveness should stay lenient under load

Prefix-cache hit rate37 · Observability

\text{prefix hit rate} = \frac{\text{rate of prefix cache hits}}{\text{rate of prefix cache queries}}

vllm:prefix_cache_hits_total over vllm:prefix_cache_queries_total; a sudden drop means more prefill per request

nvidia.com/gpu requests	integers only; 0.5 is rejected	Set in limits, never overcommitted, and all GPUs come from one node.	34
MIG slices on H100, H200, B200	7 compute slices and 8 memory slices	3g.40gb = 3 compute slices and 40 GB of memory (4 memory slices).	35
H100 80GB MIG profiles	1g.10gb ×7, 1g.20gb ×4, 2g.20gb ×3, 3g.40gb ×2, 4g.40gb ×1, 7g.80gb	L4 and L40S do not support MIG.	35
Choosing a sharing mechanism	MIG for isolation, MPS for throughput, time-slicing for convenience		35
Xid codes: cordon and drain	48 (double-bit ECC), 64 (row remap failure), 74 (NVLink), 79 (fallen off the bus), 95 (uncontained ECC)		37
Xid codes: check the application first	13, 31 and 43	119 and 120 (GSP) need a reset or reboot; 63 means schedule a reset; 45 follows another error.	37
DCGM_FI_DEV_GPU_UTIL	% of time any kernel ran	Reads close to 100% at batch-1 decode while the tensor pipe is mostly idle.	37
DCGM roof metrics	DCGM_FI_PROF_DRAM_ACTIVE (memory roof), DCGM_FI_PROF_PIPE_TENSOR_ACTIVE (compute roof)	Decode lives at high DRAM activity and low tensor activity, and that is fine.	37
OOMKilled vs CUDA out of memory	OOMKilled: host memory cgroup, exit code 137, no traceback; CUDA OOM: GPU HBM, Python traceback	Different memory, different fixes.	37

Beyond chat: reasoning, agents, images, embeddings

Workloads that stress the same engine in different places.

Time to first answer token (reasoning)31 · Reasoning & agents

\text{time to first answer token} = \text{TTFT} + n_\text{thinking} \times \text{ITL}

n_thinking = hidden thinking tokens; the engine’s TTFT only covers queueing and prefill

Agent task latency31 · Reasoning & agents

\text{task latency} \approx \text{turns} \times (\text{TTFT} + \text{short decode}) + \text{tool time}

with a warm prefix TTFT stays flat as the transcript grows; without one it grows every turn

Video token budget32 · Multimodal

\text{tokens} = \text{frames} \times \text{tokens per frame} \; (+ \; \text{audio tokens})

tokens per frame are halved if pairs of consecutive frames are merged

ExampleTen minutes at 1 fps with 360p frames is already about 90k tokens

Vector storage33 · Embeddings & rerankers

\text{vector storage} = N \times \text{dims} \times \text{bytes per dim}

N = documents (or chunks); raw vectors, before any index

Example100M documents × 1,024 dims × 4 bytes = 410 GB of float32 vectors

Reasoning vs chat request shape	chat: 2,000 tokens in, 300 out; reasoning: 2,000 in, 8,000 out	Almost pure decode, with KV that keeps growing after admission.	31
Model calls per agent task	10–50	Each call re-sends the whole transcript; prefix caching and session routing decide latency.	31
Tokens per image	256 to over 16,000, depending on model and resolution settings	Image tokens prefill and take KV cache like any text.	32
Image token schemes	LLaVA-1.5 fixed 336 px: 576 tokens; InternVL-style 448 px tiles: 256 tokens each; Qwen2.5-VL: one token per 28×28 pixels		32
Reranker candidates	rerank 50, not 500	A cross-encoder runs one forward pass per (query, chunk) pair, so cost is linear in k.	33
Shrinking stored vectors	int8 quantization 4×, binary quantization 32×, Matryoshka 256 of 1,024 dims 4×	Usually with a re-scoring pass over the top candidates to recover recall.	33