The Storage Procurement Decision That Determines Whether Million-Token Serving Is Profitable

Million-token LLM serving requires a layered KV cache strategy, with GPU HBM for active work and DRAM, NVMe, or networked storage for reusable prefixes that exceed it. Profitability depends on whether restoring those prefixes costs less than recomputing them. Storage procurement must therefore follow measured context length, prefix reuse, and concurrency, with KV format and serving topology decided before capacity and interconnects are ordered.

Why KV cache memory becomes the dominant cost driver at million-token context lengths

A KV cache holds the attention keys and values for every token the model has already processed, and it grows as generation adds tokens. The Dell Technologies KV cache survey shows how memory footprint grows in line with context length. Model parameter memory does not grow with the request. Consequently, above a model-dependent context length, KV storage overtakes parameter storage and consumes most available GPU VRAM. A large open-weights model with a standard-precision million-token cache exceeds a single high-end GPU’s capacity by a wide margin.

Capacity is only the first constraint to bind. Per-token attention reads the full KV cache at each decode step, so with a longer context, more data must move from memory to generate each new token. Concurrent requests compound the footprint, while reusable prefixes compete for the remaining space. Once that working set exceeds VRAM, the system must evict or offload it. GPUs are bought for compute, but at very long contexts they spend most of their time waiting for memory.

How bandwidth and latency penalties cascade across the storage hierarchy below GPU HBM

GPU HBM has bandwidth in the TB/s range, but CPU-GPU PCIe 5.0 has only a small share of that, roughly one to two orders of magnitude less. Moving a large cache into host memory gives it room but leaves the system with more data to transfer. A transfer over PCIe can take orders of magnitude longer than the computation it feeds, leaving expensive compute idle.

NVMe adds another discontinuity because paged GPU KV layouts map poorly to SSD access patterns. Non-contiguous blocks become fragmented reads during retrieval, so device bandwidth alone does not predict restoration time. The survey reports GPU bubbles exceeding 70% of total inference latency in tested SSD restoration cases, with layer-wise transfers pushing idle time higher. Fragmentation creates both storage inefficiency and inference delay. A miss in HBM reaches DRAM, and a miss there reaches NVMe, crossing a separate bandwidth and latency boundary each time.

Object storage introduces a semantic mismatch as well. Standard S3-compatible operations work at object granularity, while inference needs prefix-selected KV blocks in the order layers consume them. The fixed overhead of each request therefore adds time on the time-to-first-token critical path. Naive offloading moves the bottleneck into a hierarchy whose delays can accumulate before the GPU resumes useful work.

Why a cache miss is a profitability event, not just a latency event

When a reusable 1M-token prefix no longer fits in memory and is not persisted, a new request must recompute it from scratch. A well-implemented storage retrieval can be orders of magnitude faster than that recomputation. The difference spends or preserves GPU time. In multi-turn agentic workloads, growing histories are repeatedly submitted, so uncached work compounds across turns rather than appearing as a single initialization cost. Even when most requests hit the cache, misses can make a session cost many times as much as the fully cached baseline.

Building py-kvcache demonstrates that, on an H100, average requests in the Bailian production trace fall below the break-even point for external caching, while GPU memory alone retains enough prefixes. For short contexts, fast recomputation can take less time than restoration. Purchasing decisions must include the overhead of storage operations and transfers when working out whether an avoided prefill saves money overall. The break-even point depends on context length and GPU speed, and what to buy depends on which side of it the actual workload falls.

What the format of KV tensors determines about storage capacity and I/O cost before any tier decision is made

KV format sets how much data each tier must hold and move. For a full-attention cache, the sizing expression is 2 × num_layers × num_key_value_heads × head_dim × bytes_per_element × sequence_length × concurrent_sequences. The 2 counts both keys and values, while the layer count, KV head count, and head dimension describe how the model’s cache is built. Sequence length and concurrency describe the workload. Bytes per element is directly controlled by the chosen precision format.

Grouped-query attention reduces the number of KV heads relative to query heads, while hybrid architectures with sliding-window or linear-attention layers reduce the layers whose cache grows with sequence length. Moving from BF16 to FP8 halves bytes per element, cutting cache capacity demand and memory traffic per attention step in half. vLLM benchmarks report 2.6x higher burst throughput under memory-constrained load and double effective KV capacity with FP8 relative to BF16.

Sub-8-bit techniques such as TurboQuant can add quantization and dequantization overhead in compute-bound settings while reducing memory movement under high load. These techniques should be used for burst protection, with any steady-state efficiency gain left to be tested. Sizing entirely on BF16 can over-provision storage, but sizing on unvalidated aggressive quantization can miss the required accuracy. Format is a decision that precedes and constrains tier selection.

How the serving architecture — monolithic versus disaggregated prefill-decode — sets the topology that storage must fit

Monolithic serving produces and consumes KV blocks on the same node, making HBM, DRAM, and NVMe a local storage decision governed by PCIe or NVLink. Disaggregated prefill-decode serving moves KV tensors between nodes after prefill. The cache becomes a network workload as well. Storage accessibility and transport bandwidth must then fit the placement of prefill and decode workers, so architecture and procurement cannot be optimized independently.

NVIDIA’s NIXL (NVIDIA Inference Xfer Library) supports these transfers in frameworks including vLLM and NVIDIA Dynamo, using NVLink, InfiniBand RDMA, GDS/GPUDirect Storage, and TCP fallback. Transport selection changes effective bandwidth substantially. TopKV selects NVLink within the same domain, PCIe for same-node cross-domain transfers, RDMA across nodes, and TCP as fallback. Using RDMA where NVLink is available wastes bandwidth, while standard Ethernet can suffice for shorter disaggregated contexts. Disaggregation itself still needs workload validation because transfer overhead can outweigh its benefit at small batch sizes.

ObjectCache demonstrates a further topology option by making S3-compatible storage a runtime backend for reusable KV blocks. At 64K contexts, it adds only a small latency premium over local DRAM by overlapping layerwise transfer with per-layer GPU computation. With shorter contexts, less computation runs while data is transferred, so more of the transfer time adds to latency. Object storage can therefore take the place of local memory only when the architecture meets these needs.

What each storage tier actually delivers against the four axes that determine serving economics

HBM offers the highest bandwidth, lowest latency, smallest capacity per GPU, and highest cost per gigabyte. Its speed serves the decode loop directly. But it has too little room to hold the entire million-token reusable working set for a large model. It should hold the working set for active requests and the hottest prefix entries. Expanding lower tiers preserves more reusable state but does not give those tiers HBM’s access characteristics.

CPU DRAM provides an order of magnitude more capacity per node at a lower cost per gigabyte than HBM. GPU access remains constrained by PCIe bandwidth and latency. A large restoration can leave the GPU idle for hundreds of milliseconds, longer than the computation for many decode steps. DRAM therefore fits recently evicted blocks with high reuse probability. It is the warm bridge between HBM and NVMe.

Local NVMe provides tens of terabytes per node at a cost per gigabyte far below DRAM, but adds storage latency and bandwidth losses from fragmented I/O. Its natural role is cold-but-reusable system prompts, RAG corpora, and session histories beyond DRAM capacity. Layout and scheduling determine its useful performance. Tutti (2026) uses a GPU-native object abstraction for bulk transfers and direct GPU access to NVMe-resident KV data. py-kvcache shows why asynchronous direct I/O and scheduler-aware preloading, which starts reads while requests queue, are also needed to realize the tier’s benefit.

Networked object storage offers effectively unlimited capacity decoupled from compute sizing, the lowest cost per gigabyte, the highest latency, and bandwidth bounded by the fabric. Once blocks are committed, any worker can fetch them, supporting cross-session, cross-user, and cross-node reuse. ObjectCache reduces the latency penalty through consumption-ordered delivery and transfer-compute overlap on a high-bandwidth RoCE cluster. That makes the tier a good fit for stable archives with high reuse, but a poor fit for primary serving of short or low-reuse requests.

CXL remains an emerging, research-stage alternative for rack-wide prefix caching. TraCT and Beluga report substantial time-to-first-token reductions over RDMA baselines, suggesting a potential tier between DRAM and NVMe on the latency-capacity curve. Production deployments and production cost comparisons remain to be established.

Which storage tiers do long-context and million-token workloads need to be profitable?

Context length is the first branch of the decision. For short contexts, HBM may retain enough prefixes, and external caching may cost more than recomputation, as the H100 trace result illustrates. For medium contexts, DRAM offload becomes useful, with NVMe viable when contiguous writes and asynchronous preloading control I/O overhead. At long contexts, concurrent working sets push the design toward NVMe, while stable prefixes can justify object storage. FP8 or validated compression changes the capacity needed at each branch.

High reuse from stable prompts, fixed RAG corpora, and agentic histories converts persistence into repeated avoided computation. Low-reuse, one-shot queries favor aggressive eviction and smaller DRAM buffers. Mooncake, which serves the Kimi chatbot, pools CPU, DRAM, SSD, and RDMA resources for disaggregated KV caching. Its semantics-aware eviction policy retains context-cache entries while excluding ephemeral results such as web search output, increasing the global cache hit rate by 93%.

Concurrency is the final capacity test. Low concurrency may fit in HBM and DRAM even at long contexts, while high concurrency can rapidly exhaust DRAM and require NVMe or object storage to retain reusable blocks. The chosen tiers must also be reachable from the workers that need them, with disaggregation making cross-node access part of the specification. That same specification must state the format it assumes and plan for sub-8-bit quantization as burst protection while sizing the baseline without it. The framework produces a procurement specification that includes software configuration, and its tier choices must precede the hardware order.

Where current approaches still fall short and what that means for infrastructure teams procuring today

The restore-versus-recompute debate remains unresolved across workloads. Faster inference engines make recomputation a better choice for shorter contexts, while Tutti’s work on the layout is still a 2026 research result that has yet to see broad use in production. Object storage also has a limit: its fixed latency adds to the wait when computation takes too little time to hide it. Its crossover point must be measured for the model and workload. CXL also lacks the production precedent needed to make research gains a dependable purchasing assumption.

For teams procuring today, the lesson is to profile prefix reuse, context length distribution, and concurrency before specifying tiers. Treat format as an early decision, standardizing on FP8 where quality is validated and evaluating lossless compression before sizing NVMe. Match transport to topology. Disaggregated specifications should include storage accessibility and RDMA/RoCE bandwidth at 32K+ contexts, while CXL should remain contingent on available hardware and workload-validated benchmarks.

The storage design should also support adaptive eviction rather than fixed tier assignment, following the survey’s finding that no single technique dominates across deployment scenarios. Benchmark before procurement and again after deployment. Those tests must use the actual KV layout, transfer schedule, and workload, since general-purpose vendor storage results alone will not cover them. Storage for LLM inference is a first-class engineering problem, and its acceptance criteria must measure serving economics as well as device performance.