KV Cache Eviction Policies and GPU Utilization Tradeoffs
Choosing which tokens to keep matters far more than how you organize the eviction structure.
KV cache eviction policies trade GPU memory for assumptions about what future queries will need. The right policy is not the one that discards the most tokens. It is the one that matches recency, salience and coverage to the workload, then combines eviction with quantization, storage offload and prefix reuse to raise utilization without unacceptable accuracy loss.
Why GPU memory is the binding constraint in long-context LLM serving
For an L-layer, H-head, d-dimensional model, KV cache memory scales as O(LHnd), where n is context length. Under standard multi-head attention, the cache doubles in size when the context doubles. As context grows, the cache overtakes model weights as the dominant memory consumer. Grouped-Query Attention reduces the constant factor, raising the ceiling on context length.
With million-token context windows, the cache for a single user running a large-scale model needs more HBM than a flagship GPU has. Full-cache residency is therefore no longer an available default. The serving system must decide what remains in HBM, what leaves it and what information it can afford to discard.
What eviction actually does to a running inference system
Prefill builds the KV cache in one parallel pass, while decoding repeatedly reads it as tokens are generated. Eviction after prefill discards representations that have already incurred computation cost and changes what every later decoding step can access. Sliding-window methods such as StreamingLLM preserve initial tokens and a fixed recent window while discarding the rest. That rule is straightforward, but it ignores salience and can degrade long-context quality.
Top-k methods instead use an importance score to rank tokens and choose which to keep. H2O uses accumulated attention across query positions, while SnapKV selects high-attention token clusters during prefill and concatenates them with a short observation window. FastGen combines special tokens, punctuation, recent tokens and top-k selections to suit the patterns in each head. These methods differ in what evidence they accept as a reason to retain a token.
Discarding a KV pair is also distinct from moving it to another storage tier. A discarded pair can be recomputed later, while a stored pair can be reloaded. Which path costs less depends on context length, hardware and how often the pair is reused. PCIe 5.0 provides only a small fraction of HBM bandwidth, so offload must be weighed against prefill recomputation.
The scoring problem and how each policy decides which tokens are important
CacheCraft’s May 2026 scorer-versus-structure decomposition isolates the importance signal as the decisive design choice in its evaluation. Using FRC in place of a baseline scorer adds +67.2 RULER points, while a change to the chunk-selection structure adds only ≈ 0.1 points. A better retention structure cannot compensate for a scorer that selects the wrong information.
H2O’s cumulative attention measure requires access to attention scores that conflict with FlashAttention’s fused kernel design. SnapKV’s recent observation window is fast, but its scores are query-dependent and can miss tokens that matter over a long range. CacheCraft reports that single-statistic baselines also change rank across models, with ChunkKV leading on Llama-3.1-Instruct at aggressive compression ratios and ExpectedAttention strongest on Qwen3 in some settings.
FRC, or Feature-Rich Compression, combines three normalized signals with fixed weights. The weight for local attention received is 0.55, the weight for neighborhood attention density is 0.30, and the weight for KV-head maximum salience is 0.15. These signals capture complementary properties, namely temporal locality, spatial coherence and head-specific salience. CacheCraft finds the combination through LLM-guided program evolution, using program search to design the policy.
The per-head uniformity flaw that most policies share
Most policies divide the cache budget uniformly across attention heads, although heads attend to different token subsets and serve different roles. Equal budgets can therefore over-evict heads carrying disproportionate long-range information. The 2026 ICML paper “Rethinking KV Cache Eviction via a Unified Information-Theoretic Objective” frames the objective as jointly maximizing structural diversity and query relevance, neither of which a uniform budget directly optimizes.
Eviction scores overlap across heads and layers, causing similar tokens to be removed repeatedly and collapsing unique-token coverage. Its cross-head and cross-layer modules redirect selected heads toward tokens not retained elsewhere and promote globally important tokens missed in earlier layers. Using an information-bottleneck framework, K-VEC shows in theory that less coverage limits the mutual information between input and output.
K-VEC reports up to 10.35 points of improvement across 16 LongBench subsets under the same eviction rate and memory constraint, with particularly large gains at low budgets. PyramidKV partially addresses heterogeneity by assigning more cache to earlier layers. It does not, however, address cross-head redundancy within a layer.
When eviction degrades accuracy beyond what benchmarks advertise
Coverage failures become operationally serious because eviction losses vary sharply by input. Capacity-bound requests can degrade severely regardless of policy, while dilution-prone requests tolerate selective retention better. Most policies cannot tell which regime a request is in. An average benchmark score therefore cannot show that a policy is safe for every request in production.
The PAGE paper in 2026 documents accuracy falling from near-perfect to zero on Mistral NIAH-MK3 under plain eviction. A gating mechanism keeps accuracy at 89% and gives the same protection to realistic code completion tasks. Cache reuse introduces another failure mode when independently-prefilled chunks are placed at arbitrary positions without cross-chunk attention. A 2026 preprint evaluating KV reuse methods finds substantial CacheBlend degradation on HotpotQA for Llama 3.1-8B, attributed to coarse-grained recomputation that does not adapt to per-document attention distributions.
These cases show the consequences of aggressive compression or policies poorly matched to their inputs. Before selecting a policy, teams need to distinguish context-heavy workloads, including long-range reasoning and multi-document QA, from context-light workloads such as summarization or code generation with short prompts. The same retention rule can be acceptable in one setting and destructive in another.
Quantization as a different kind of bet, shrinking tokens rather than discarding them
Quantization preserves all tokens while reducing the bits used to represent them. FP8 KV caching roughly halves the footprint relative to BF16 and reduces bytes read during decoding. It is already widely used in production and makes the system faster when KV reads are bandwidth-bound.
The accuracy tradeoffs differ across formats. INT8 KV caching costs 1.5–3 points on long-context retrieval benchmarks, compared with 0.3–0.7 points for FP8, a range within noise for most production workloads. NVFP4, introduced with the Blackwell architecture, saves 75% versus BF16 but carries larger quantization error. Moving to 4 bits is especially hard because the error grows nonlinearly. KIVI reaches 2-bit quantization by using per-channel quantization for keys and per-token quantization for values, an aggressive choice requiring workload-specific accuracy evaluation.
Quantization and eviction compose multiplicatively, so FP8’s roughly 2× compression can reduce the amount of token removal needed for a given footprint. MLA-based architectures, including DeepSeek variants, reduce cache memory requirements structurally before a policy runs. Even that architectural reduction does not make storage tiers irrelevant at the longest context lengths.
What happens when even compressed caches outgrow GPU HBM
When compressed caches exceed HBM capacity, useful extra storage needs enough bandwidth to serve them. HBM delivers TB/s, CPU-GPU PCIe 5.0 provides a small fraction of that bandwidth, and NVMe SSDs are another order of magnitude slower again. A very large model answering a short question about a long document may need to transfer tens of gigabytes over PCIe. That transfer is costly relative to a short decoding sequence, but it can still beat recomputing prefill during a long document session.
The crossover shifts with model size, context length and hardware generation. Transfer efficiency also depends on the path through the system. The 2026 Tutti paper identifies GPU stalls when fragmented cache layouts produce massive numbers of tiny random SSD I/Os that the low-parallelism CPU cannot service quickly enough. Requests above 8 KB cannot bypass that CPU path because NVMe PRP list allocation requires privileged CPU code unavailable to GPU programs. Tutti removes CPU intervention from the critical data and I/O control paths, allowing coarser I/O without CPU fallback.
NVIDIA’s ICMSP, announced at CES 2026, takes a platform approach to NVMe cache offload. It uses BlueField-4 for hardware-accelerated placement and combines the DOCA framework, Dynamo KV cache offload engine and NIXL for accelerated sharing across AI nodes. NVIDIA claims up to 5× greater power efficiency than traditional storage and up to 5× higher tokens-per-second. Together, ICMSP and Tutti show the architectural direction of removing CPU work from the critical path for storage-tier KV access.
How disaggregated serving systems assign eviction and storage decisions across nodes
Disaggregated prefill-decode serving adds network transfer and cross-request reuse to the retention decision. A block can be valuable because many requests share it, because retransferring it is expensive or because it has high attention salience. vLLM’s PagedAttention shares physical pages across requests with common prefixes and removes per-request contiguous reservation. This supports high utilization under bursty traffic, although page-level eviction is coarser than token-level selection.
SGLang’s RadixAttention makes prefix reuse explicit through a radix tree keyed by token sequence. A common prefix is computed once and reused across requests, so evicting it removes a benefit shared by many consumers. Architecture also constrains offload options. SGLang 0.5.12 includes HiSparse for GPU-to-CPU offloading for DeepSeek Sparse Attention models, including DeepSeek-V3.2, GLM-5.1 and DeepSeek V4.
Moonshot AI’s Mooncake, presented at FAST '25, separates the cache layer from the serving engine and trades more storage for less computation. LMCache provides an enterprise-scale cache layer between serving engines and storage tiers, integrating with vLLM and SGLang so placement and eviction can be managed independently of engine logic. SplitZip in 2026 addresses transfers between prefill and decode workers over RDMA-capable interconnects. Its GPU-friendly lossless compressor exploits floating-point exponent redundancy in BF16/FP8 tensors before transfer, shifting the binding constraint from raw bandwidth to transfer latency. Eviction consequently becomes a distributed decision about reuse, placement and movement.
Mapping eviction policy choices to deployment scenarios
For long-context single requests such as document QA, legal review and code analysis, coverage is the binding concern. FP8 quantization, or lower precision where accuracy permits, should precede eviction as the first footprint reduction. Coverage-aware K-VEC or multi-signal FRC is preferable to a single-statistic scorer when selective retention is needed. MLA-based architectures reduce the structural burden before these choices arise.
For high-throughput serving with short-to-medium contexts, shared prefixes change the economics. RadixAttention benefits many requests by retaining one common prefix, while PagedAttention combines page sharing and continuous batching for good GPU utilization. Its coarser granularity is more acceptable when context is moderate. NVMe offload through ICMSP or Tutti-style GPU-direct paths can extend effective capacity without aggressive in-cache eviction.
Multi-turn conversations make recency a more defensible signal because recent turns are often more relevant to the current query. Early turns can still contain critical context, which a pure sliding window silently discards. StreamingLLM keeps both the initial tokens and a recent window to help with this. Accuracy-critical legal, medical and correctness-sensitive code workloads warrant a different balance, with FP8 as the lower-risk reduction and eviction gated or minimal after workload characterization.
For disaggregated serving, SplitZip-style lossless compression changes transfer costs, while LMCache-style layers allow placement and eviction to be tuned independently of the engine. These deployments must account for the cost of retransferring remote blocks as well as retaining local ones. Combining those policies across HBM, CPU DRAM, NVMe and RDMA-connected remote storage requires both enough storage capacity and the bandwidth to keep up with the chosen policy’s I/O profile.
