KV Cache Storage Cost in Long-Context Serving

KV cache storage cost is not some fixed tax the industry pays for long context. It's a direct output of a formula, one built from a handful of architectural variables that any infrastructure engineer can name, measure, and, in most cases, control. That distinction matters because as of September 2026, the IntuitionLabs report from that month found that long-context serving stretching from tens of thousands of tokens into the millions has turned KV cache into the largest and fastest-growing line item in production GPU memory budgets.

The cache exists for a simple reason: without it, every decoding step would recompute the key and value tensors for every prior token, and generation cost would grow steeply with sequence length. So instead, the system stores those tensors once and reuses them. But storage isn't free. Each cached tensor sits in VRAM for the entire life of a request, growing token by token, layer by layer, and it never shrinks until the request ends. On a hosted API, the provider eats that cost and folds it into pricing. On owned hardware, the cache competes directly with model weights for the same scarce pool of HBM, and that competition is exactly where teams get surprised. Plenty of infrastructure groups size their VRAM around parameter count alone, only to discover at serving time that the cache, not the weights, is what's eating their memory budget.

The formula: every term that drives KV cache size

The math behind KV cache size is: 2 × num_layers × num_kv_heads × head_dim × seq_len × bytes_per_element × active_sequences.

Every term here maps to something concrete, and once you see the mapping, the optimization options become obvious. The factor of 2 covers the fact that you're storing two separate tensors, keys and values, not one. num_layers means every attention layer in the model keeps its own cache, so depth compounds the cost directly: a 40-layer model pays 40 times over what a single-layer model would. num_kv_heads × head_dim gives you the KV width per token per layer, and this is the term that attention variants like MQA, GQA, and MLA are built to shrink. seq_len is prompt tokens plus generated tokens combined, the one term an operator actually controls at runtime, and the one that balloons in long-context work. bytes_per_element is the quantization lever: 2 bytes for BF16 or FP16, 1 byte for FP8 or INT8, roughly half a byte for INT4. And active_sequences is just batch size, multiplying everything else, so every concurrent user adds cost linearly.

IntuitionLabs finds that KV cache size tracks almost entirely with KV head count and layer depth, not with total parameter count. That's the whole reason GQA, MQA, and MLA all target the head dimension rather than trying to shrink the model itself. KV cache memory scales linearly with context length and batch size, not quadratically. The O(n²) cost people usually cite belongs to the attention computation itself during prefill or training. Persistent cache storage is a different animal, and it's linear. That's good news and bad news at once: it won't explode as fast as the compute cost does, but at the context lengths agentic workloads now demand, linear growth alone is enough to overwhelm a GPU.

What the formula produces at real context lengths and batch sizes

Run the formula on a real model and the numbers stop being abstract fast. Llama 3.1 70B at 128K context, running in BF16, is roughly 40 GB of KV cache per request. Compare that against the model weights themselves, which need around 140 GB in FP16, and you land on an uncomfortable fact: at 128K tokens, the cache for a single request is already approaching the size of the model's own FP16 weights.

Batch size makes this worse in a hurry. A 70B-class model using standard multi-head attention at just 8K context, with a batch size of 32, needs around 640 GB of KV cache on its own, the Introl.com guide found. Break that down per token and you get roughly 2.5 MB per token at that architecture, a number that sounds small until you multiply it by tens of thousands of tokens and dozens of concurrent sequences.

Above 128K tokens, KV cache memory exceeds parameter memory on most architectures. Above 128K tokens, KV cache memory exceeds parameter memory on most architectures. Push out to 1M tokens with naive multi-head attention in FP16, and cache size hits roughly 135 GB, more than the weights of the model generating it. At that point the context window is a capacity constraint, not a recall feature. It's the dominant line item on the VRAM budget, full stop.

This is also why data-center GPUs keep getting more memory. The H100 shipped with 80 GB, the H200 with 141 GB, and Nvidia's Grace Blackwell Superchip carries 372 GB, the IntuitionLabs September 2026 report found. Cloud pricing tracks the same pressure: on-demand rates for these chips ranged from $3.99 per GPU-hour for the H100 to $5.99 for the H200 as of September 2026, the same report found, though AWS cut on-demand GPU pricing by up to 45% in a change made back in June 2025. Baselines move fast in this market, which is exactly why memory efficiency isn't a nice-to-have. It appears directly on the invoice.

Why long-context and agentic workloads hit the formula hardest

Agentic loops are where this problem gets sharp edges. These workloads routinely push tens of thousands to hundreds of thousands of tokens into the model as input, tool outputs, retrieved documents, prior turns, all while generating a comparatively tiny number of output tokens. At that kind of input-to-output ratio, prefill compute dominates total GPU time, not decoding.

That's where prefix caching earns its keep. A 90% KV cache hit rate means the server skips the overwhelming majority of prefill work it would otherwise redo, cutting time-to-first-token substantially and slashing effective compute cost per request by a large share. That's not a marginal win. It's the difference between a usable agent loop and one that stalls on every tool call.

But provisioning for advertised context windows carries its own trap. The RULER benchmark found that only about half of evaluated long-context models hold up at 32K tokens, well under many vendors' advertised maximums, IntuitionLabs's citation of that work confirmed. A KV cache sized for a 128K or 1M-token advertised window is a real, billable memory allocation regardless of whether the model can actually use that much context productively. Teams need to validate usable context length, not just take the vendor's number and provision against it.

Anthropic's own documentation has a name for the underlying failure mode: "context rot," the gradual decline in accuracy and recall as token count climbs. This phenomenon is one mechanism behind the gap between advertised and effective context length. And disaggregated serving setups, where prefill and decode run on separate workers, introduce additional coordination overhead worth accounting for in long-context deployments.

PagedAttention: the memory management layer every production stack already runs

Before PagedAttention existed, inference systems wasted somewhere between 60% and 80% of allocated KV cache memory to fragmentation and over-allocation, as IntuitionLabs reported. That waste came from a simple design flaw: systems reserved contiguous memory sized for the maximum possible sequence length even when a request never used all of it.

PagedAttention fixes this by borrowing an idea straight from operating system design. It treats KV cache memory like virtual memory: fixed-size blocks, allocated on demand, scattered non-contiguously across physical memory. Nothing needs a giant contiguous reservation anymore. PagedAttention brought memory waste under 4% and throughput improvements of 2 to 4 times, per IntuitionLabs.

By 2026, this isn't an optional optimization anyone debates. vLLM, SGLang, and TensorRT-LLM all ship PagedAttention by default, the Digital Applied engineering guide found. It's the substrate everything else sits on top of. The only real configuration decision left to engineers is block size, with 16 or 32 tokens being the most common choices depending on workload.

Prefix caching builds directly on this foundation. When multiple requests share a common prompt prefix, whether that's a system prompt, a RAG template, or a repeated agent instruction, the KV blocks for that shared prefix can be reused instead of recomputed. vLLM implements this through Automatic Prefix Caching using prefix hashing, while SGLang uses a radix tree structure through RadixAttention, a different mechanism aimed at the same outcome. In agent loops, RAG pipelines, and multi-tenant SaaS deployments, production hit rates of 60% to 85% are achievable, and at a 90% hit rate the effective compute cost per request drops dramatically, as noted earlier.

Architectural compression: how GQA, MQA, and MLA reduce the formula at its source

If PagedAttention manages the memory you already have, GQA, MQA, and MLA shrink how much memory the model demands in the first place. Each trades some amount of attention expressivity for cache savings, and they're not interchangeable, since the amount of quality given up varies a lot.

Multi-Query Attention pushes the trade the furthest. Every query head shares a single key and value head, which on a 32-head model gets you up to 32 times compression. A quality regression of 1 to 3 points on most tasks accompanies this trade-off, steep enough that it's not widely used in frontier 2026 models outside latency-sensitive serving contexts.

Grouped-Query Attention is in the middle and, as a result, dominates outside the DeepSeek family in 2026, delivering roughly 4 to 8 times compression over standard multi-head attention. Llama 3 70B uses GQA with 8 KV heads against 64 query heads, and that architecture choice drops per-token cost from roughly 2.5 MB under an MHA baseline down to about 0.3 MB at FP16 (S6). It's also supported across every major inference stack without special tooling, which is a big part of why it won the popularity contest.

Multi-head Latent Attention, DeepSeek's contribution, takes a different route entirely: instead of storing full key and value tensors, it stores a low-rank compressed latent projection, yielding 7 to 14 times compression, the Digital Applied April 2026 guide found. DeepSeek's own paper reports over 90% KV cache reduction against a comparable dense architecture. DeepSeek V2, V3, and V4 all build on MLA, while competing model families largely stay on GQA's 4 to 8 times compression. MLA is the specific reason V4-Pro can serve a 1M-token context economically at all: combined with FP8, it takes KV cache down from roughly 135 GB under naive MHA and FP16 to around 8 GB, a reduction near 94%, as the Digital Applied guide's own table shows.

Sliding Window Attention takes a completely different angle on the problem. Instead of compressing the head dimension, it restricts some or all layers to a fixed local attention window, capping the seq_len term for those layers no matter how long the total sequence gets. The logic holds up reasonably well in practice: most next-token predictions lean heavily on nearby context anyway, and deeper layers still propagate signal forward through the network, so local windows can produce coherent output even without full-sequence attention.

None of these approaches are mutually exclusive. Production systems increasingly stack GQA or MLA together with sliding windows and quantization, treating each as a separate lever on a separate term in the same formula.

KV cache quantization: compressing bytes_per_element and its accuracy cost

Quantization goes after the bytes_per_element term specifically. It's a storage-time operation: the attention matmul itself still runs in BF16 or FP16, but what sits in memory between computations gets compressed.

FP8 halves memory against FP16 or BF16, and that 50% savings translates into 30% to 50% throughput gains, mostly because it lets you run a larger batch size within the same VRAM ceiling, the Digital Applied April 2026 guide found. The accuracy cost is modest, 0.3 to 0.7 points on long-context retrieval benchmarks, close enough to noise for most production workloads. FP8 has a real scar from April 2026: on a 128K needle-in-a-haystack retrieval task, vLLM's FP8 accuracy collapsed from a 91% BF16 baseline down to just 13%, traced back to imprecise FP32 accumulation inside Tensor Cores. The fix shipped that same month (S1). A second finding from that incident matters just as much for planning purposes: for models using sliding-window attention layers, the FP8 decoding-latency slope came in nearly identical to BF16, at 96% of the BF16 slope, and the memory savings didn't translate into faster decoding below roughly 700K tokens. Benchmark the specific architecture before assuming FP8's memory win becomes a speed win.

INT8 delivers a comparable memory cut to FP8, but the accuracy tax runs higher, 1.5 to 3 points on long-context, multi-needle retrieval tasks (NIAH-2), the Digital Applied guide found. On the hardware side, a vectorized CUDA kernel implementation of GPU-accelerated INT8 quantization can reach speedups many times over CPU baselines, while keeping reconstruction error under 0.004 and attention score error under 0.1, even at large head dimensions, the research behind this implementation found.

NVFP4 goes a step further still, but only on the latest generation of hardware from one vendor. It cuts memory by another 50% beyond FP8, taking a Llama 3.1 70B request at 128K context from roughly 42.9 GB in BF16 down to about 21.5 GB in FP8 and then to roughly 10.7 GB in NVFP4 (S1). The hardware constraint is not a suggestion: NVFP4 needs Blackwell chips like the B200 or RTX 5090, and on H100 or A100 hardware it will either error out or behave incorrectly, depending on the vLLM version in use. That's a check engineers need to make before flipping the switch, not after.

Beyond that sits the research frontier: approaches like KIVI, KVQuant (aimed specifically at very long context inference), and MiniKV have pushed toward 2-bit KV cache storage. These remain research-stage rather than production-ready. A reasonable heuristic for teams choosing among the options: start with FP8, since it has the broadest hardware support and smallest accuracy hit, reserve INT8 for cost-sensitive deployments willing to absorb a bigger accuracy regression, and treat NVFP4 and sub-2-bit methods as hardware-specific or research-specific tools rather than defaults.

Advanced eviction and memory reuse: workload-aware strategies beyond LRU

Basic eviction policies, LRU, FIFO, LFU, all share one blind spot: they treat every cached KV block as equally valuable, regardless of what's actually in it or how likely it is to get reused. A block holding a shared system prompt used by every request in a multi-tenant deployment gets evicted under the same rules as a block from a one-off query nobody will ever repeat. That's a mismatch between how the eviction policy behaves and how real workloads actually use the cache, and it's the reason more workload-aware strategies, ones that weigh reuse probability and request structure rather than just recency, have become the next frontier for teams trying to squeeze more out of the same HBM footprint.