Scoring Inference Vendors on Whether They Actually Solve the Storage Problem in Long-Context Serving

Long-context LLM serving infrastructure needs a four-tier KV cache architecture spanning GPU HBM, CPU DRAM, local NVMe, and shared persistent storage. For long context windows, capacity alone is insufficient. Vendors must demonstrate that their I/O paths, cache policies, and cross-replica retrieval avoid unnecessary recomputation without degrading accuracy, with reproducible public benchmarks as the standard of proof.

Why long context turns inference into a storage problem

KV cache grows with layers, attention heads, head dimension, sequence length, and batch size. At FP16/BF16 precision, those dimensions together make the per-token cost high. A single long-context session on a large model can approach tens of gigabytes before concurrency enters the calculation. Each concurrent session needs more space. Extended-context and very-long-context windows are production workloads as of 2026, so this is an operating constraint.

When KV state exhausts GPU memory, the engine must evict and recompute or move state to slower storage. Neither choice is free. Recomputation is frequently not cost-competitive in long-context agentic workloads, while offloading puts retrieval on the path to serving the next request. Tool calls sharpen the problem because a session can release GPU capacity while suspended, then require its prior state on return. Without surviving KV state outside the GPU, that return incurs full prefill again. A vendor’s architecture must handle the physics of the tiering hierarchy beyond merely supporting KV offloading.

The tiering hierarchy that any real solution must navigate

GPU HBM holds hot KV blocks for active sessions because attention needs its bandwidth. CPU DRAM holds warm completed blocks, extending capacity at microsecond access latency. Local NVMe provides a cold tier for historical prefixes. Shared persistent network storage makes cache state available beyond a single worker and supports disaggregated prefill/decode designs. These tiers serve different purposes, and their order-of-magnitude differences in bandwidth and latency constrain where blocks can live.

The CPU-to-GPU link is the first sharp bottleneck. PCIe 5.0 has only a small fraction of HBM bandwidth, so moving data out of GPU memory makes room but slows access. CPU DRAM is nevertheless the highest-leverage upgrade for most deployments because it extends cache capacity many times over. The vLLM offloading blog identifies the native connector API introduced in vLLM 0.11.0 as using CPU DRAM as its primary offload target. Its LRU/ARC cache also provides a staging area for deeper tiers.

Beyond DRAM, the access mechanism becomes increasingly important. DUAL-BLADE documents page-cache thrashing under memory pressure when file-backed mmap handles autoregressive KV access. Shared storage introduces a different mismatch because S3 delivers objects or byte ranges while the engine consumes KV state layer by layer. Hardware alone cannot resolve either mismatch. Depending on admission, eviction, and prefetch scheduling, a retained prefix can become a useful hit or a retrieval that costs more than recomputation.

Why bandwidth claims without I/O path specifics are not engineering answers

A claim of NVMe offloading does not identify which path moves the data to the GPU. The standard filesystem path traverses VFS, the filesystem, the block layer, and the device driver, adding software overhead that prevents full NVMe bandwidth utilization. File-backed mmap also exposes decoding to page-cache thrashing. A drive’s bandwidth specification therefore does not establish the bandwidth available to KV retrieval.

GPUDirect Storage removes the CPU bounce buffer through direct PCIe P2P transfers between NVMe and GPU HBM. That improvement does not make every access pattern efficient. GDS is optimized for large sequential I/O over block-based abstractions, whereas KV staging involves fine-grained, high-frequency access that can produce high latency and limited bandwidth on NVMe-backed GDS. Virtualized deployments add another restriction because IOMMU interference conflicts with the direct memory access GDS requires. A claim about a standard KVM cloud must state that limit.

The practical guideline is to use io_uring for inference transfers and GDS for pre-training and fine-tuning checkpointing, where large sequential I/O dominates. Scheduling still determines how much transfer time reaches the request. Building py-kvcache shows that transfer granularity, intermediate memory use, and the timing of transfers all affect external-cache performance. Its asynchronous preloading starts disk reads while requests remain queued, overlapping retrieval with compute. The admission decision is therefore setup-specific, since short prefixes or fast GPUs can make recomputation faster than external retrieval.

The accuracy failure that vendor benchmarks almost never show

KV Cache Offloading for Context-Intensive Tasks evaluated modern offloading schemes on Text2JSON, which requires dense structured extraction from long context. The study found significant performance degradation on both Llama 3 and Qwen 3. Throughput alone would not expose this failure. Neither would tests that measure only perplexity on standard held-out text, because the task needs many specific details to be retrieved even when the language model performs well overall.

The study identified two causes of the degradation. Low-rank projection of keys loses the fine-grained positional signal that dense lookup needs, while unreliable landmark selection can leave out the tokens needed for the answer. This is distinct from lossy compression. Even without quantizing KV data, an offloading policy can discard high-salience tokens through its selection process. Storing the retained blocks exactly does not repair that omission.

A system that performs well on summarization or single-hop QA may consequently degrade on RAG, agentic document parsing, or multi-hop extraction. Those are precisely the workloads that can justify retaining long context. A lossless storage claim is insufficient without a lossless access pattern. Context-intensive accuracy tests establish whether offloading preserves the information the workload actually needs.

What lossless BF16 compression contributes — and what it cannot fix

BF16 contains statistical redundancy that lossless compression can exploit without approximating KV values. Its exponent field uses only a fraction of its theoretical entropy, implying a theoretical lossless compression ratio around 1.51×. GAAC, a lossless KV compression and serving system described in arXiv 2026, achieves 1.405–1.407× lossless BF16 KV compression across five tested models. The gain is real but modest. It reduces storage demand and transfer volume while preserving the represented data.

For comparison, lossy quantization with FP8 or NVFP4 achieves two to four times compression at the cost of output quality. Lossless compression is the appropriate choice when that quality must not degrade. GAAC also provides less additional compression on FP8 KV caches than on BF16, so its benefit depends on the representation entering the compression layer. A stack that already quantizes captures less additional benefit from lossless encoding.

Compression that preserves quality belongs in the offloading stack, but the stack still needs an adequate fabric. Fewer bytes still cross the same constrained links. Compression therefore complements io_uring and RDMA rather than answering the I/O-path and network questions on their behalf. A vendor must show how much smaller the data gets and how its transfer architecture moves the data that is left. An attractive compression ratio does not establish that runtime retrieval can keep pace with inference.

Persistence and sharing across replicas: the tier most vendors quietly skip

Local caching stops helping when the required state disappears with a worker or remains inaccessible to the replica resuming a session. Agentic and multi-turn workloads expose this boundary whenever execution suspends and later resumes elsewhere. Without shared persisted KV state, the new replica pays full prefill again. Persistence is therefore part of recompute avoidance. It is not merely a way to expand a worker’s cache.

Prefix KV blocks are immutable once generated, making them suitable for content-addressable storage and deduplication on object storage. Because the blocks do not change, S3-compatible storage can serve as a persistence tier, but having room for objects alone does not make runtime retrieval efficient. The engine needs layers in consumption order. Standard S3 access over conventional network transports is too slow for latency-sensitive runtime operations in this design, requiring RDMA transports to close the gap. Protocol and transfer scheduling must make stored objects available for layerwise consumption.

ObjectCache explores that bridge by co-designing retrieval and transfer scheduling so the storage server delivers KV data in the order the GPU consumes it. Transfers overlap with compute across concurrent requests. This shows how the architecture could work, but with no open-source integration at publication, it does not support recommending the product for production use. The shared tier must disclose whether it requires proprietary transport or works over standard RDMA with an S3-compatible API, and whether adoption requires code changes or only configuration.

What MLPerf Storage v3.0 results reveal about which storage systems can keep up

MLCommons announced MLPerf Storage v3.0 results on September 1, 2026, adding KV cache and vector database workloads to training and checkpointing tests. The suite also added access to S3 object storage alongside POSIX. Nineteen organizations sent in results, and roughly one-sixth of the submissions used the S3 access layer. This creates a public comparison framework for storage claims. Its coverage includes Unet3D and RetinaNet training, Llama 3-style checkpointing across model scales, KV caching, and vector database tests.

At the largest model scale for checkpointing, the top distributed systems had write bandwidths in the hundreds of GiB/s, which grew in a predictable way as more nodes were added. Closed KV cache submissions on a distributed system tested throughput in the tens of thousands of tokens per second for Llama 3.1 models at multiple scales. They included both storage-only and storage-plus-memory modes. Those distinctions prevent checkpoint bandwidth from standing in for evidence about KV serving.

Power efficiency adds another architectural test. For the same workload, the gap between median and top results was more than an order of magnitude, so node count alone cannot explain the results. MLCommons also noted its intent to align future KV cache rounds with MLPerf’s agentic inference benchmark as that work matures. Buyers should judge the results against the workload, hardware configuration, and access layer they plan to use. MLPerf results, or equivalent reproducible public benchmarks, provide the appropriate evidence for KV cache and checkpointing claims.

The five scorecard dimensions that separate real solutions from marketing claims

First, the architecture must support all four tiers and expose its admission and eviction policies. Stopping at DRAM or local NVMe leaves persistence and fleet-wide reuse unresolved. Policy must also account for retrieval break-even points. A fixed rule that treats every prefix hit as beneficial can underperform when short-context recomputation is faster than external retrieval.

Second, the vendor must identify which I/O path serves KV transfers and whether it matches their access pattern. VFS/mmap, io_uring, GDS, and custom kernel-bypass paths are not interchangeable descriptions of the same capability. KV access is fine-grained, random, and write-then-read. In particular, a GDS claim must disclose virtualization limitations rather than presenting direct access as universally available.

Third, KV state must survive a worker’s lifetime and be retrievable by a different replica. The vendor must specify the persistence tier, its transport requirements, and how it delivers layerwise data to the engine. Standard RDMA and an S3-compatible API address portability without requiring a proprietary transport. Configuration-level integration also answers a different concern from persistence itself, namely how much serving code must change.

Fourth, vendors must establish whether context-intensive accuracy survives offloading, including dense extraction, structured output, and multi-hop QA. Perplexity and summarization results do not substitute for those tests. The answer must distinguish lossless encoding from selection policies that can omit high-salience tokens. Lossless BF16 compression keeps data exact only if attention can still access it.

Fifth, public, reproducible results must support the performance claims on a comparable deployment. A commitment to benchmarking is not a result. MLPerf Storage v3.0 or an equivalent public methodology must identify the hardware configuration, workload, and S3 or POSIX access layer behind the numbers. Checkpointing evidence cannot substitute for KV cache evidence, and storage-plus-memory results cannot silently stand in for storage-only performance. The score rests on what the architecture demonstrably delivers under the buyer’s conditions.