Why KV Cache Explosion Is a Unit Economics Crisis Before It Is an Engineering Problem

Long-context LLM serving infrastructure needs to be sized for model weights and per-request cache memory at the intended concurrency. KV cache explosion is a unit economics crisis because the growing need for memory sets the cost floor of inference. Efficient attention architecture, cache reuse, quantization, storage tiering, and, at sufficient scale, disaggregated serving are best judged by their effect on cost per request.

Why flat per-token pricing masks a cost structure that scales with context depth

A flat per-token rate treats token N as though it costs the same to serve as token 1. That assumption conceals the cost of context depth. A later token belongs to a longer sequence, whose retained state occupies more GPU memory for longer and displaces other revenue-generating requests.

The key-value cache, or KV cache, is that retained state. It avoids recomputing past token representations during generation, but its memory footprint grows with the sequence. A long-context request does not merely consume more tokens than a short one. It also leaves less room for other requests while it runs, reducing the amount of concurrent work the same hardware can support.

Consequently, flat pricing can systematically underprice long-context requests relative to their serving cost. An operator can recover those costs across a mixed workload, but short-context revenue then subsidizes heavier users. The margin depends on the request mix. For an enterprise deployment without external API revenue, the same mismatch appears in budgeting, where an average token rate obscures the infrastructure cost of longer conversations.

How KV cache memory actually scales — the formula and what it produces in practice

KV cache memory follows the formula 2 × num_layers × num_kv_heads × head_dim × seq_len × batch_size × bytes_per_element. The terms for the architecture set the cache footprint before serving starts, while sequence length, batch size, and precision govern how much it grows during use. The relationship with context length is linear. Doubling the context doubles the cache for each active request, even though the model’s weights stay the same.

Model weights have a fixed cost that requests share, while each request needs its own cache. At an extreme context length, the large-model example given here has roughly 135 GB of KV cache alongside approximately 140 GB of weights. The context length and architecture are not specified for that example, so it is not a sizing rule for a named model. It nevertheless illustrates how a single request’s retained state can approach the size of the shared model itself.

Longer contexts therefore change what it means for a model to fit on hardware. Loading the weights establishes only the starting requirement. It does not establish how much useful serving capacity remains once requests begin to accumulate state. For contexts measured in millions of tokens, that distinction becomes central to infrastructure planning because cache growth continues with every added token rather than stopping at the model’s fixed weight allocation.

Concurrency is the silent multiplier that turns a per-request cost into a cluster-sizing crisis

Concurrency multiplies the per-request allocation across active users. The requests compete for GPU high-bandwidth memory, or HBM, which is both scarce and expensive. Even with shared weights and an identical underlying model, a deployment sized around one successful long-context request is a poor guide to the capacity needed for many simultaneous requests.

Context length determines each request’s cache size, and concurrency determines how many of those allocations must coexist. At moderate long contexts with many concurrent users, the collective cache can exceed what a single-node GPU configuration can hold. Fragmentation overhead, activations, and tensor-parallel runtime allocations consume additional memory.

Enterprise RAG and multi-turn agents bring this constraint into ordinary operation when long contexts coincide with high concurrency. The expensive case becomes the workload the infrastructure must sustain, and the cluster budget accounts for both the aggregate cache requirement and the model’s weight size.

Why model architecture is the highest-leverage cost decision — and it is made before any serving choice

The per-token KV footprint varies by more than an order of magnitude across architectures serving similar capability levels. That difference is fixed before serving software enters the picture. Multi-Head Attention, or MHA, uses as many KV heads as attention heads and produces the largest footprint in this comparison. An operator selecting such a model for long contexts accepts a memory cost that later software can manage but cannot fully erase.

Grouped-Query Attention, or GQA, reduces that footprint by letting multiple query heads share fewer KV heads. Llama 3 70B and most 2024+ open-weight models use this approach. Multi-Head Latent Attention, or MLA, goes further through low-rank KV compression. The DeepSeek-V2 paper describes MLA as its attention mechanism, and the DeepSeek-V3 Technical Report states that the later model also adopts it. The saving originates in model design.

DeepSeek’s structurally different KV economics help explain its inexpensive long-context API offering, but the broader lesson is not a recommendation to select that provider. For operators choosing open-weight models, per-token KV footprint should be a first-class criterion for long-context workloads. Capability and serving cost meet at the attention architecture, and changing that cost structure after deployment requires switching models.

How PagedAttention and prefix caching turn memory waste into recoverable throughput

Once architecture is fixed, memory management determines how much of the remaining capacity becomes useful throughput. Naive allocation can waste HBM through fragmentation across variable-length requests. PagedAttention addresses that waste with a small per-token compute overhead in exchange for near-full use of available memory across concurrent batches. Its financial return is recovered capacity, allowing more of the hardware already purchased to serve requests.

Prefix caching recovers a different resource, namely work repeated across requests with identical beginnings. Long system prompts, reference documents, and few-shot templates can otherwise have their KV state computed independently for every request. vLLM’s automatic prefix caching hashes incoming prefixes and reuses cached KV blocks when it finds a match. SGLang’s RadixAttention extends sharing to tree-structured prompts across more complex multi-call patterns.

Together, these techniques are a low-risk starting point for production. Neither requires a model change or an accuracy trade-off, and both convert avoidable waste into lower serving cost. Their economic roles remain distinct, with paging recovering fragmented capacity and prefix caching recovering repeated work.

KV quantization and tiered storage as cost levers, with their real accuracy trade-offs

KV quantization reduces the bytes needed for cached tensors by moving from BF16 to lower-precision formats. The vLLM team’s April 2026 benchmarks put FP8 per-token cache memory at 54% of the BF16 baseline, with accuracy losses of 0.3–0.7 points on long-context retrieval benchmarks. For many production workloads, the measured regression falls within noise, making quantization an attractive economic choice after the foundational allocation and reuse techniques are in place.

Agentic and multi-hop reasoning workloads can build up precision losses over many tool calls, so even a small FP8 regression may be too much. There is still no accepted production benchmark that measures how much quality users lose with quantization at scale. Quantization is therefore not a universally safe substitute for more memory, even when its capacity benefit is clear.

Storage tiering addresses the remaining capacity problem by moving cache beyond GPU HBM. vLLM with LMCache supports CPU DRAM, local NVMe, and remote destinations including S3 and Redis. Transfers introduce a different constraint, which is GPU stall time. The Tutti system, a 2026 academic work integrated into vLLM, provides a GPU-native object abstraction for bulk KV transfers and demonstrates near-zero GPU stall time while saturating NVMe SSD bandwidth. Quantization and tiering can compose because they change different parts of the cost structure.

DRAM alone has a limited retention horizon at high concurrency. Even roughly 2 TB retains only about five minutes of KV cache under those conditions, making NVMe necessary for persistent prefix caching at scale. Capacity is only half the storage requirement. HBM bandwidth, DRAM throughput, NVMe IOPS, and RDMA latency determine whether offloading frees expensive memory without starving the GPU. When cache often moves between memory tiers, that movement needs to be planned as part of the architecture as well as handled by background storage tasks.

How should long-context serving separate prefill and decode at scale?

Prefill and decode place different demands on hardware. Prefill processes a large contiguous token block in parallel and is compute-bound, while decode reads a paged KV cache one token at a time and is memory-bound. Running both on the same GPU forces one resource allocation to serve unlike bottlenecks. Disaggregation separates them into dedicated pools so that prefill nodes can be sized for compute throughput and decode nodes for memory bandwidth and capacity.

The separation makes cache transfer a critical path. RDMA and NVLink provide the high-bandwidth interconnects that make this arrangement feasible, while a slow transfer path can leave it worse than a combined deployment. Disaggregation therefore changes more than scheduling. It makes the network and cache-storage hierarchy part of the direct serving path, joining compute allocation, memory capacity, and data movement in one cost decision.

AMD and Crusoe’s MLPerf Inference v6.1 submission provides related hardware-level evidence in the most recent benchmark cited here, with results released September 16, 2026. The submission used separate attention kernels for compute-bound prefill and memory-bound decode over a paged KV cache, substantially improving attention performance. Lower-precision quantization of DeepSeek R1 also freed up more space for cache on the same hardware. The large-GPU submission recorded the highest total LLM throughput ever achieved in an MLPerf submission. This supports phase-specific optimization, though it is not by itself evidence for the economics of separate node pools.

Disaggregation needs KV cache routing, transfer scheduling, and fault tolerance across resources that are managed separately. Smaller teams may not be positioned to build or operate that infrastructure. At a large enough scale and with pressure to meet long-context service-level requirements, though, gains in throughput can make the extra work worthwhile because each phase can get resources that fit its bottleneck.

Choosing optimizations by financial leverage rather than technical sophistication

The practical decision criterion is cost per request at the workload’s actual context length and concurrency. Technical elegance is not a proxy for return. For operators selecting a model, attention architecture offers the largest initial leverage because it sets the per-token footprint before any serving optimization. For an existing deployment, the first action is to verify that PagedAttention and prefix caching are active before evaluating more demanding changes.

FP8 quantization offers high leverage for general-purpose inference when its measured quality regression is acceptable, while precision-sensitive agentic workloads require a different judgment. Storage tiering becomes essential when long-context concurrency or persistent prefix reuse exceeds available memory, but pays off only when transfers avoid GPU stalls. Disaggregation makes sense at a scale where throughput and service-level requirements are worth the cost of separate resource pools and the work of running them.

Workload shape determines the order of investment. A short-context API at high concurrency does not have the same priorities as an enterprise platform running multi-turn agents over extended contexts. A software feature may also fall short of its theoretical saving if NVMe scheduling or RDMA transfer is the bottleneck. Purpose-built AI storage earns its cost where those physical constraints would otherwise prevent cache reuse, tiering, and disaggregation from delivering useful throughput. The purchase is justified by lower serving cost.