KV Cache Cost Control as a Storage Tiering Problem Not a Model Optimization Problem

Long-context LLM serving infrastructure needs a tiered KV cache architecture. Serving long contexts at scale requires placing active blocks in GPU HBM, extending capacity through CPU DRAM, and using NVMe or shared storage where retrieval beats recomputation. KV cache cost control is therefore a storage placement problem governed by eviction policy, tier bandwidth, and retrieval latency.

Why KV cache stops being a model problem and becomes a data placement problem

Quantization, sparsification, and attention approximation can reduce the per-token footprint without providing a destination for state that still exceeds GPU memory. The capacity gap remains an infrastructure constraint. On an H100 serving a large long-context model, a single user's KV cache can take up a large share of the GPU's entire 80 GB HBM, or even need more than that. Several concurrent users therefore require more cache capacity than that GPU provides.

KV cache size scales linearly with context length, batch size, number of layers, and model width. Dell Technologies' survey documents context windows growing from tens of thousands to hundreds of thousands and, in some cases, millions of tokens. Longer contexts compound the capacity demands of concurrency. Once the working set outgrows HBM, the architecture must determine which state stays resident, which state moves elsewhere, and whether retrieving it costs less than rebuilding it.

What infrastructure does long-context serving need across the four memory tiers?

GPU HBM holds the active, hot KV blocks because it offers the highest bandwidth and shortest access time. It is also the most expensive capacity. CPU DRAM provides the first extension, with microsecond-range access and a lower cost per GB, although transfers to the GPU are limited by PCIe. A large DRAM allocation can preserve more prefixes without making their restoration free.

Local NVMe extends the hierarchy to cold sessions and long-tail prefixes. It offers inexpensive capacity and sub-millisecond access, but its usefulness depends on drive parallelism and write endurance as well as read speed. Distributed or object storage adds another trade-off. A network hop increases retrieval cost, while shared storage enables reuse across replicas, persistence, and warm-starting. Erasure coding can reduce storage overhead at this tier compared with replication.

Red Hat's OpenShift AI 3.5 benchmarks evaluated CPU DRAM offloading, CPU plus local NVMe offloading, and CPU plus CephFS offloading against H100 GPUs. The secondary storage options both retained the host-memory tier. In these setups, DRAM both holds state that is reused often and stages transfers between GPU memory and deeper storage. The resulting architecture must be assessed as a complete path.

The restore-versus-recompute crossover: when tiering pays and when it doesn't

The break-even test compares the time needed to restore cached KV state with the time needed to recompute its prefix. Which takes less time depends on model block size, storage latency, transfer bandwidth, and context length. The restoration path includes movement into GPU memory, so even a DRAM hit incurs a cost that must be weighed against prefill on the particular model and GPU.

Understanding Bottlenecks for Efficiently Serving LLM Inference With KV Offloading identifies the cached-to-prefill token ratio as a critical variable. At high ratios, PCIe transfers can account for the vast majority of latency while GPU compute remains underused. The constraint shifts from computation to data movement. Transfers within HBM are orders of magnitude faster than DRAM-to-GPU transfers, so host-memory bandwidth limits what the architecture can do and adds much more than a small overhead.

The py-kvcache study shows the reverse case. For short prefixes or fast GPUs, external retrieval can cost more than recomputation, especially when GPU memory already retains enough prefixes. The same study also shows that scheduling changes the crossover at long contexts. Scheduler-aware preloading starts disk reads while requests are waiting, so retrieval can run alongside other work and cut the loading delay compared with loading without it. Break-even analysis therefore depends on both when transfers enter the schedule and how quickly a device completes them.

How the host-centric data flow in vLLM's tiered offloading works in practice

vLLM's tiered KV cache offloading, available since v0.22, routes data through host CPU DRAM before propagating it to secondary tiers. During offload, asynchronous DMA moves KV chunks from GPU memory to the host. GPU memory is then freed before secondary writes begin. Those writes use the host copy without going back to GPU memory, so the GPU can reclaim capacity without waiting for state to be stored elsewhere.

The host LRU/ARC cache is checked first, and a miss triggers queries to secondary tiers in their configured order. The first tier holding a chunk serves it and promotes it asynchronously to host memory. A request need not have one storage source. Different chunks can arrive from different tiers, while the host-centric design enables just-in-time GPU memory allocation.

A canonical host layout makes that data portable. It uses one block per layer, with all KV heads gathered into a single contiguous region, so nodes with different GPU types, attention backends, or parallelism configurations can share KV data without remapping. Secondary tiers run as single processes per vLLM instance using CPU-side I/O libraries and never touch GPU APIs. Filesystem and object-storage tiers also use content-addressed naming, so token sequences that are the same map to the same key. As a result, instances that share a mount point can reuse KV data without any extra configuration.

What the I/O engineering stack must do to close the bandwidth gap between tiers

An NVMe device's rated bandwidth does not establish how quickly an inference server can feed the GPU. Three choices shape the delivered result. They are whether I/O submission is asynchronous or blocking, whether CPU memory sits in the data path, and how transfers overlap with inference. The py-kvcache results make the scheduling component concrete, because preloading changes performance without changing the underlying storage tier.

For submission, io_uring offers lower latency than libaio and competitive IOPS for small random NVMe operations, matching the access pattern of KV cache workloads. GPUDirect Storage takes a different approach to data movement. It bypasses CPU memory on the NVMe-to-GPU path and is better suited to throughput-dominant training and fine-tuning workloads. A host-centric cache design and a direct storage path impose different requirements on the surrounding software.

Distributed serving adds transport and routing decisions. NVIDIA Dynamo uses NIXL to transfer KV state between prefill and decode workers through RDMA, NVMe-oF, and S3, separating transport from compute. DualPath (2026) adds a second loading route for agentic workloads, allowing decoding engines to retrieve KV state and forward it to prefill engines over RDMA. This redistributes network load across retrieval paths. NVIDIA's ICMSP, announced at CES 2026, uses BlueField-4 DPUs to manage cache movement off the GPU compute path. Across these designs, the common engineering goal is to keep data movement from consuming or delaying the resources needed for the forward pass.

What the right storage backend for KV cache tiering actually requires

A storage backend for KV tiering must handle high-bandwidth reads and writes simultaneously, with low tail latency at the block level. Prefetch and eviction operate concurrently, so a system must sustain retrieval while accepting newly offloaded state. Distributed tiers also need S3-compatible access, while workloads that cannot compromise output quality require lossless handling of BF16 tensors.

MLPerf Storage v3.0 makes these requirements measurable. Its results, released September 1, 2026, came from the first benchmark round to treat KV cache as a storage workload by reading and writing the inference cache at the same time. The 1,623.4 GiB/s KV cache read bandwidth result applies specifically to the measured submission.

The same round added an S3 object storage data layer alongside the POSIX layer that was already there, and roughly one-sixth of submissions used S3 access. Object storage also has a concrete cache design in ObjectCache (2026, FAST '26). It stores fine-grained, hash-addressed KV chunks in S3-compatible storage and extends S3 with layerwise delivery semantics. This makes object granularity part of the retrieval design. Backend suitability therefore depends on block-level latency, mixed read/write performance, integration with cache delivery, and the amount of inexpensive capacity available.

Which workloads cross the threshold where storage tiering delivers the largest gains

Storage tiering delivers its largest gains when the KV working set consistently exceeds HBM, prefix reuse is high, and restoring state costs less than repeating prefill. Multi-turn RAG, multi-codebase agents, and large long-context multi-turn deployments have these traits. Their shared prefixes repeatedly cycle through eviction and reuse. But when average contexts are short, a fast H100 can fall below the break-even point that py-kvcache found. For that regime, the lesson is to make external caching an admission-controlled option.

The OpenShift benchmarks distinguish two capacity cases. For large-model, long-context agentic workloads whose working sets exceeded CPU-tier capacity, CPU offload alone provided meaningful throughput gains, while adding NVMe or CephFS nearly doubled that gain. Smaller models had different results when their working sets fit in DRAM. CPU offload gave them most of the gain, and deeper tiers added little more.

Under high concurrency, CephFS sustained cache reuse while CPU-only configurations increasingly fell back to recomputation. Cross-node sharing was the differentiator. Session migration extends that capability, because a new GPU instance can resume decode from stored KV state, avoiding a repeat of full prefill. That supports disaggregated serving and load balancing without quality loss, moving the benefit beyond the capacity or throughput of any single node.

Tiering KV cache as an infrastructure architecture decision, not a per-node tuning choice

Once cache state can move across replicas, procurement must account for GPU count, memory capacity, and the full retrieval path. For NVMe, capacity is only the starting requirement. Continuous, write-heavy KV traffic calls for datacenter-class drives with high drive writes per day and multiple drives for bandwidth parallelism. At the distributed tier, erasure coding reduces storage overhead compared with replication while maintaining similar durability. At petabyte scale, that lower storage overhead has a large effect on cost.

Evaluation must follow the same architectural scope. MLPerf's KV Cache workload provides a public baseline for simultaneous reads and writes, while the September 1, 2026 round included 11 first-time submitters and covered both POSIX and S3 access layers. Public results make performance claims open to scrutiny. They do not remove the need to measure the serving workload's restore-versus-recompute crossover, since a storage benchmark cannot determine how much prefix reuse an application produces. The two measurements answer different procurement questions, namely whether the backend can sustain cache traffic and whether that traffic saves GPU work.

Adoption should rest on performance that others can reproduce and compare publicly, with more to back it up than isolated vendor benchmarks. Open-source systems let others inspect how they work, while tiered offloading and right-sized hardware make it possible to build the hierarchy today. Storage deserves the same rigor as compute architecture. For long-context serving, that means specifying placement, eviction, transport, and retrieval as infrastructure decisions before treating KV cache cost as a model optimization task.