Model Weights & Loading

The cold-start breakdown showed that a new replica spends much of its start-up time moving weights. This chapter covers that part in detail. Three questions decide how it goes:

• What exactly are you loading? The weights alone do not make a working model. The tokenizer, chat template and generation defaults have to match them.
• In which format? Some formats can be memory-mapped and sliced per GPU. Others have to be executed to be read, and some are compiled for a single GPU type.
• Where do the bytes come from? A 140 GB checkpoint takes about 8 minutes to download from the internet and about 25 seconds to read from local NVMe. If twenty replicas download it at the same time, those numbers diverge further.

What is inside a model repository

A Hugging Face-style model repository for Llama 3.1 70B Instruct holds about 140 GB, nearly all of it weights. The small files matter just as much:

Sharding exists for practical reasons: downloads can resume and run in parallel, and no single file goes beyond filesystem or hub limits. Sharding also means a model on disk is a set of files that must all come from the same revision. Half-updated caches are a real failure mode.

Formats: what the bytes look like

safetensors is now the default for serving. The format is deliberately simple. Click through the tensors of one shard below, then switch to the pickle view to compare:

The format has three parts: an 8-byte little-endian integer giving the header length, a JSON header that maps each tensor name to its dtype, shape and data_offsets, and then the raw tensor bytes. Because the offsets are known before any data is read, a loader can memory-map the file and copy only the bytes it needs. With tensor parallelism, each rank reads only its slice, and the OS faults in only the pages that rank touches. Reading needs no Python object construction and runs no code.

The other formats you will meet:

Where the weights live at run time

On Kubernetes, the weights have to come from somewhere at pod start. There are six common patterns:

• Hub at start-up. The engine downloads into its cache directory on every start. This is fine for a laptop and fragile in a cluster: every pod pays the full download, egress is shared, and a hub outage or rate limit becomes your outage.
• ReadWriteMany PVC. Download once into an NFS, Lustre or CephFS volume and mount it read-only in every pod. Simple to run, but every replica now reads through the same filer.
• Node-local NVMe cache. A DaemonSet or pre-pull Job writes the weights to each GPU node’s local disk, and pods mount them with hostPath or a local volume. This is the fastest option, and you have to manage cache eviction and disk space yourself.
• Object storage, streamed. The weights live in S3, GCS or an S3-compatible store. vLLM’s --load-format runai_streamer reads tensors concurrently into host buffers and copies them to the GPU while later tensors are still downloading, with no local copy.
• OCI artifacts. The weights are packaged as an image layer. KServe Modelcars (storageUri: oci://…) and Kubernetes image volumes (beta and on by default from 1.35, GA in 1.36) mount them read-only. You get digests, signing, registry mirroring and the node’s image cache for free. Every model change is a new build and push.
• Internal mirror. A pull-through proxy or artifact repository in front of the hub, with an allow-list. It keeps the hub workflow while removing the dependency on the internet.

Bandwidth is not the only factor. What matters is bandwidth per replica when several replicas start together, because scale-out events arrive in bursts:

There is no single right answer. The trade-offs shift with fleet size, update frequency, model size and whether the cluster can reach the internet:

A pattern that works at most sizes is one immutable source of truth plus a cache near the GPU. The source of truth is a mirror, a bucket or an OCI artifact addressed by digest or commit hash. The cache is a node-local copy that pre-pull jobs warm before the autoscaler needs it.

How tensor parallelism changes the read

With TP = 4, each of four worker processes needs a quarter of every large matrix. How the loader gets there matters:

• With safetensors + mmap, each rank maps the shard files and copies out only its slice. Column-parallel slices are contiguous. Row-parallel slices are strided, a piece of every row. Each rank moves about 1/TP of the bytes to its GPU, but the files and page cache are shared on the node. If the page cache is cold, the disk sees roughly one full read in total.
• With pickle, every rank unpickles the full state dict into CPU memory and then throws away three quarters of it. Host RAM briefly needs TP × the checkpoint size.
• Pre-sharded checkpoints (one file set per rank, saved after a first load) and TensorRT-LLM engines turn this into a plain sequential read per rank. They are tied to that exact TP layout.

The host-to-GPU copy runs over one PCIe link per GPU (Gen5 x16, about 64 GB/s per direction peak; loaders reach a fraction of that). Adding GPUs to a replica therefore speeds up the last stage, until host memory bandwidth or the loader’s CPU threads become the limit.

Pod settings that make loading reliable

spec:
  containers:
    - name: vllm
      image: vllm/vllm-openai:<pinned-version>
      args: ["/models/llama-3.1-70b-instruct",          # local path, not a hub name
             "--tensor-parallel-size", "4",
             "--load-format", "safetensors"]
      env:
        - { name: HF_HUB_OFFLINE, value: "1" }            # never reach for the internet at start
        - { name: HF_HOME, value: /cache/hf }              # tokenizer/config cache on a volume
      resources:
        limits: { nvidia.com/gpu: 4, memory: 200Gi }        # host RAM: see below
        requests: { memory: 200Gi }
      volumeMounts:
        - { name: weights, mountPath: /models, readOnly: true }
        - { name: shm, mountPath: /dev/shm }
        - { name: hf-cache, mountPath: /cache/hf }
  volumes:
    - name: shm
      emptyDir: { medium: Memory, sizeLimit: 16Gi }        # NCCL / multiprocess workers

• /dev/shm. Containers get 64 MiB by default. TP workers and NCCL use shared memory for inter-process communication, and too little shows up as hangs or bus errors at start-up. An emptyDir with medium: Memory fixes it. Its contents are tmpfs pages and count against the container’s memory limit.
• Host memory during load. GPU memory is not the only thing that runs out. Loaders hold CPU copies of tensors, pinned staging buffers and, with pickle, full state dicts per rank. If the memory limit is sized for steady state, the pod is OOMKilled halfway through loading, which looks like a crash loop with no Python traceback. Measure the peak RSS during a load and set the limit above it.
• Offline mode. HF_HUB_OFFLINE=1 makes any accidental hub access fail fast instead of hanging on a firewall. It catches a hub name left in the args, or a tokenizer fetched separately from the weights.
• Cache directories on volumes. Point HF_HOME (or the engine’s download directory) at a volume, never the container’s writable layer. Otherwise each restart downloads again, and the node’s ephemeral storage fills up until the kubelet evicts pods.
• Probes. Size the startup probe for the slowest realistic load, including the 20-replica case from the calculator above. The cold-start chapter covers the probe design.