Why Long-Context LLM Serving Is a Storage Architecture Decision Before a Model Decision

Long-context LLM serving is a storage architecture decision before a model decision. Serving 128K-token or longer contexts at scale requires a plan for KV cache placement across GPU HBM, CPU DRAM, NVMe, and, where reuse justifies it, networked storage. How much those tiers hold and their restoration bandwidth set the limits on concurrency, latency, and cost before teams can benefit from model capability or serving-stack tuning.

Why KV cache state grows in proportion to context length, and what that means for memory

Per-token KV cache size is the number of transformer layers multiplied by KV heads, head dimension, bytes per element, and two for the key and value tensors. Sequence length then multiplies that footprint, while concurrent requests add their own state. Transformer attention keeps a key and value tensor for every token at every layer for as long as the session lasts. Longer contexts thus make the cache larger even when model weights stay the same.

Grouped-query attention architectures such as Llama 3 use fewer KV heads than full multi-head attention, though how much fewer depends on the model. That shrinks the per-token footprint without removing its dependence on sequence length. At long contexts, KV state routinely overtakes weights as the dominant VRAM consumer, reversing the usual starting assumption for hardware sizing. Reasoning workloads add another pressure during generation. KARA research shows that long chain-of-thought traces keep the cache growing through decoding, which makes it harder to bound how much memory concurrent requests will need when scheduling them.

How the memory hierarchy responds: GPU HBM, CPU DRAM, NVMe, and networked storage as a forced cascade

Once KV demand exceeds HBM capacity, the system must evict state, recompute it, or spill it into a slower tier. Spilling is often cheaper than recomputing at current context lengths, so a storage hierarchy becomes a consequence of the workload. Active KV blocks occupy HBM, warm overflow moves to CPU DRAM, and colder state moves to local NVMe. Shared networked storage can extend this hierarchy for persistence or cross-node reuse. Capacity planning consequently extends beyond the GPU itself. Multi-terabyte DRAM configurations on an 8-GPU inference node are defensible as the cheapest available extension of effective VRAM.

LMCache formalizes this approach by serializing KV blocks to configurable backends, including local NVMe, Redis, and remote object stores. It also lets subsequent requests reuse shared prefixes. Moonshot AI’s Mooncake, the serving platform for Kimi, extends the approach across a cluster. It separates prefill and decoding clusters and pools CPU, DRAM, SSD, and RDMA resources into a shared KVCache store. The bandwidth gap between tiers governs the cost of bringing cached state back into service.

The bandwidth gaps between tiers are the governing constraint on serving latency

For long-context document QA on a large model, transferring a KV cache over PCIe can take far longer than the GPU computation needed to generate the output tokens. The GPU can spend most of that interval idle. HBM supplies bandwidth in the terabytes-per-second range, while PCIe 5.0 CPU-to-GPU transfers deliver only a small fraction of it. Moving a large cache from CPU DRAM can therefore be orders of magnitude slower than reading the same state from HBM. Faster computation does not remove that transfer delay.

LM-CXD research identifies CPU cache contention between I/O and inference threads, mandatory host-DRAM staging before GPU transfer, and NAND latency as separate contributors. Replacing NAND with DRAM behind the same block interface still leaves significant overhead. The interface itself imposes a substantial penalty.

Vrije Universiteit Amsterdam’s py-kvcache research finds that transfer granularity, intermediate memory use, and the point at which transfers enter the request schedule also affect performance. A cache can have ample capacity and a fast device yet still restore state inefficiently if its transfers interact poorly with inference. Restoration is consequently a scheduling problem as well as a bandwidth problem. A request’s latency covers the whole path, with an individual device’s peak transfer rate accounting for only part of the delay.

Why recompute-vs-store is an economic and architectural decision, not a default

A cache hit outside HBM is not automatically a saving. The engineer’s decision depends on context length, GPU speed, hit rate, and workload cost, because lookup and restoration can cost more than repeating prefill. The py-kvcache study finds this break-even problem for short prefixes and fast GPUs. H100 serving workloads can gain little from external caching when they have high in-VRAM hit rates and a working set that fits.

The comparison changes with long contexts and concurrency at production scale. Recomputing a 128K-token prefill is routinely more expensive in wall-clock time and GPU-hours than restoring its KV state from NVMe. Prefill computation scales quadratically with sequence length, while reload time depends on the volume of state and available I/O bandwidth. Reuse remains the deciding condition, however. If most entries are never read again, persisting them consumes write endurance and bandwidth without recovering that expense through later hits.

Workload characterization must therefore precede the retention policy. Storage preserves valuable computation when prefix hit rates and context reuse are high and accumulates unused state when they are low. Enterprise RAG pipelines, multi-turn agents, and shared-document QA have structurally high prefix reuse, tilting the choice toward restoration. Low-reuse workloads do not inherit that conclusion. The resulting policy determines how much DRAM and NVMe capacity is useful, how much write traffic must be provisioned, and how restoration enters the request schedule.

Precision format and compression are storage throughput levers, not just accuracy trade-offs

Bytes per KV element multiply every capacity and transfer requirement in the hierarchy. BF16 and FP16 use two bytes per element, FP8 uses one byte, and FP4 in NVFP4 uses half a byte. NVFP4 reduces cache memory by 75% compared with BF16, with research characterizing accuracy penalties as near-negligible for most inference tasks. A smaller representation also reduces the volume that must cross each tier boundary, directly connecting model-side quantization to storage throughput.

The University of Bayreuth’s Benchmarking KV-Cache Optimizations shows why compression ratio alone is an inadequate selection criterion. KIVI4 provides the most stable quality across models, while SnapKV delivers the strongest long-context throughput. CaM produces large gains on selected QA workloads but is substantially more sensitive to workload type.

SplitZip targets disaggregated serving and outperforms NVIDIA’s nvcomp library codecs as baselines for ultra-fast lossless KV compression. Applied before an NVMe write or RDMA transfer, this approach reduces bytes in flight while preserving the cached values. The codec still needs to understand the format. BF16 and FP8 have different bit distributions and compressibility profiles, so general-purpose compressors leave storage and transfer savings unrealized.

CXL-SSDs and RDMA as the emerging answers to the block-interface and transport bottlenecks

LM-CXD research from Sogang University, Samsung Electronics, and ETRI finds that a stock CXL-SSD is no faster than an NVMe SSD for KV cache workloads. A hardware replacement alone is not sufficient. CXL-SSDs expose NAND-backed capacity through a byte-addressable, memory-semantic interface, removing the block I/O path, CPU staging copy, and associated cache contention. Yet the serving engine knows which KV chunks it needs, while a generic device prefetcher does not. That semantic gap limits the benefit of the new interface.

The research lets the device see KV chunks as I/O units to close this gap. Its co-designed system achieves time to first token within 1.5× of local DRAM on average across five LLM models. Standard approaches such as GPUDirect Storage and host-side prefetch can move or overlap block-interface costs without removing them. CXL-based access addresses the interface, while RDMA provides transport for shared KV resources across nodes. Both are parts of a serving architecture whose policies for moving state and scheduling requests must work together.

NVIDIA’s ICMSP, announced at CES 2026 and later rebranded CMX at GTC 2026, offloads cold KV blocks from HBM to NVMe over BlueField-4 DPUs. It uses cuFile for GPUDirect NVMe transfers, keeping data movement off GPU streaming multiprocessors. NVIDIA cuObject, released in 2026, extends direct transfers to S3-compatible object storage without CPU staging. These products address distinct portions of the restoration path.

MLPerf Storage v3.0 institutionalizes KV cache I/O as a first-class infrastructure benchmark

MLCommons announced MLPerf Storage v3.0 on September 1, 2026, adding a KV Cache test for LLM inference cache reads and writes. The test recognizes KV caching as a widely adopted technique for autoregressive transformer inference. In simulated multi-turn conversations, the test writes a context once and reads it again and again, with transfer sizes from 64 MiB to roughly 3 GiB. The median workload includes tens of thousands of contexts totaling many terabytes of data, bringing both cache volume and repeated retrieval into the evaluation.

The benchmark also adds a layer for access to S3 object storage. NVIDIA’s AIStore submission environment includes end-to-end RDMA over Ethernet and NVIDIA GPUDirect Storage, reflecting a target architecture that serves training, checkpointing, and inference-cache workloads. This broadens what storage evaluation needs to capture. An infrastructure choice must account for access paths, workload behavior, and the capacity of an NVMe device or networked tier. Practitioners can now compare storage systems against a reproducible KV workload rather than relying entirely on internal proxies for inference performance.

What infrastructure is needed to serve 128K or longer contexts at scale?

Infrastructure teams should begin with prefix hit rate, context length distribution, session reuse patterns, and expected concurrency. Those inputs tell teams whether they have a high-reuse RAG pipeline or a low-reuse single-turn service before they choose hardware for either. The next decision is where retained state will live and how quickly it must return to the GPU. That determines DRAM provisioning per node, NVMe drive count, and whether block NVMe or a co-designed CXL-SSD path fits the workload.

The same workload model should guide whether to separate prefill and decoding, whether KV transfer volume justifies RDMA, and which compression format belongs before each tier boundary. These choices constrain subsequent GPU provisioning, inference-framework selection, and precision tuning. Concurrency depends on the full usable KV hierarchy. Efficient NVMe offload and restoration can support substantially more simultaneous long-context sessions on the same GPU hardware than a design that treats HBM as the only cache tier. The practical target is capacity that the system can restore within its serving latency requirements.

Other architectures such as state space models can remove the need for the KV cache, but the research does not show that they are used at scale in frontier production deployments. They do not remove the present infrastructure decision. For teams building transformer serving systems now, storage placement, transfer bandwidth, and reuse policy must be settled before model selection or serving-stack tuning can be judged against production requirements.