BF16 and FP8 Compression Strategies for KV Cache Storage

KV cache memory, not model weights, decides how many users a long-context deployment can actually serve at once. Most teams still size their fleets around weight storage, and that habit is the wrong one once context length climbs past a few thousand tokens. Every attention layer, for every token a model has already processed, has to keep a key tensor and a value tensor in memory so the next token doesn't force a recompute of attention across the whole sequence. That's the entire point of the cache: trading memory for compute. The bill isn't negotiable once you fix a model and a context length, either. It scales directly with the number of layers, the number of KV heads, the head dimension, the sequence length, the batch size, and the bytes used per stored value.

Run the arithmetic on Llama 3.1 70B at BF16, 128K context, one concurrent user: 2 × 80 × 8 × 128 × 131,072 × 1 × 2 comes out to roughly 42.9 GB, just for that one request's cache. Switching to FP8, the same request drops to about 21 GB. At that context length, the cache stands on its own as the single largest memory line item for the request, and here's the detail that should reorder how infrastructure gets planned: a 43 GB KV cache is bigger than the same model's weights stored at INT4, roughly 35 GB, though it still comes in under the weights at FP8, around 70 GB https://arxiv.org/pdf/2607.08057. Model compression used to be the whole conversation in this field. At long context, the cache itself now outweighs a quantized model, and any capacity plan that still treats weights as the dominant cost is working from the wrong number.

The structural differences between BF16, FP8, and general-purpose data formats

BF16 packs 16 bits, the same 8-bit exponent field as full FP32 plus a 7-bit mantissa, trading range for precision. That's the whole design tension in one line. Wide range, thin precision.

That tradeoff trips up general-purpose compression, and it's why a plain zip-style codec is the wrong tool for a KV tensor no matter how well it works on text or images. Codecs built for text, image data, or generic binary blobs assume bit patterns that are close to uniform or already entropy-rich, with no obvious structure left to exploit. BF16 breaks that assumption completely. Its wide exponent field produces value distributions that are anything but uniform, and a codec that doesn't know it's looking at a floating-point exponent compresses it poorly. Format awareness has to come before compression, not after. Guessing at the bit layout gets you nowhere.

For KV cache specifically, E4M3 dominates for a concrete reason: the dynamic range of K and V activations is bounded by the softmax, so the format's extra range headroom goes to waste while precision stays the scarce resource. E4M3 trades range for precision on purpose, and that happens to be exactly the trade this tensor needs.

FP8 as the current default quantization target: accuracy, throughput, and hardware considerations

The core trade is simple: moving KV cache storage from BF16 to FP8 cuts memory in half. What's changed recently is the posture around that move. It used to read as aggressive, something you'd reach for only under memory pressure. Recent vLLM guidance frames FP8 KV cache quantization as the default starting point for long-context deployments generally.

The accuracy cost backing that recommendation is small enough to make the trade close to a free lunch. On standard long-context, retrieval-heavy benchmarks, quality loss typically stays under half a percentage point https://vllm-project.github.io/2026/04/22/fp8-kvcache.html.

On A100, which has no FP8 Tensor Cores, the KV tensors are dequantized before attention computation, so the saving is memory-only with no throughput gain from the quantization itself. On H100, the story changes in a way that matters for anyone chasing latency, not just footprint. vLLM stores the KV tensors in FP8 and runs the entire attention computation, both the QK and ScoreV matrix multiplies, natively in FP8. That's a throughput win stacked on top of the memory win, and it only exists on hardware with native FP8 Tensor Cores.

Blackwell pushes the same idea one generation further. On B200 and RTX 5090, the --kv-cache-dtype nvfp4 flag gets a further 50 percent reduction on top of FP8's savings. BF16 and FP8 have structural properties (exponent bias, narrow mantissa, non-uniform value distributions) that make general-purpose compression fail and require purpose-built quantization, residual coding, and storage strategies tailored to how attention tensors actually behave at scale.

The quantization spectrum below FP8: INT8, INT4, and 2-bit strategies with their accuracy trade-offs

FP8 isn't the floor. Below it sits a spectrum of increasingly aggressive schemes, each buying more headroom at some cost to fidelity, and the honest takeaway is that most of that spectrum isn't worth touching unless a specific constraint forces the issue.

INT8 KV cache quantization maintains accuracy, while INT4 shows a slight loss.

KVQuant takes a more surgical approach: quantize keys per-channel, since that's where outliers in the key tensor concentrate, and quantize values per-token instead. The granularity gets matched to how each tensor's statistics actually behave rather than applying one blanket rule to both, which produces a mixed-precision scheme that pushes down toward 2-bit while holding fidelity. KIVI goes further still, running tuning-free asymmetric 2-bit quantization on the KV cache with no fine-tuning step required, and that marks something close to the practical floor of the lossy spectrum as it stands today.

TurboQuant 4bit-nc delivers up to 3.4× KV cache capacity, with 1 to 4 points of accuracy degradation on most benchmarks, a meaningful memory-for-throughput trade. Against plain FP8, though, the other variant doesn't offer a meaningful advantage. Not every scheme further down the bit-width ladder beats FP8 by default. Some of them just cost more than they return, and picking one because it has a lower bit count on the label, without checking what it costs on your benchmark, causes accuracy problems that appear three weeks into production. TurboQuant comes in two variants with very different outcomes.

Lossless KV compression: what structural properties of BF16 and FP8 make it tractable

Lossy quantization isn't the only lever available, and it took a while for lossless compression of KV cache to become practical for a specific reason: general-purpose lossless codecs were never built for this job. They're designed for offline weight compression, they typically run on CPUs, and many lean on variable-length coding schemes that can't keep pace with how fast KV tensors get produced during prefill.

SplitZip was built to close that gap. It's a GPU-friendly lossless compressor for KV cache transfer that preserves the tensors bitwise and slots into existing serving frameworks without touching model execution. It works because both BF16 and FP8 share a structural fact: exponent values aren't spread uniformly across a tensor, and some exponent patterns show up far more often than others. SplitZip assigns fixed-length codes to the frequent exponent values and routes the rare ones through a separate escape stream, keeping the whole coding scheme fixed-length and therefore GPU-friendly, unlike traditional variable-length entropy coders.

Lossy and lossless compression don't compete for the same job, and treating them as alternatives misses how they're actually deployed. They stack: one narrows the representation, the other squeezes the redundancy out of what's left.

DFloat11 takes a different lossless route entirely, achieving 70 percent size compression https://arxiv.org/pdf/2607.08057.

Compression strategy across the storage hierarchy: GPU memory, NVMe offload, and network transfer

Compression decisions don't look the same at every tier of the storage stack, because each tier has its own bandwidth and latency profile, and that profile, not habit or convenience, is what should decide the trade-off. The path runs from GPU HBM, fastest and smallest, down through CPU DRAM, then local NVMe, then out to networked storage or remote nodes.

At the GPU HBM tier, FP8 (or nvfp4 on Blackwell) is the right call for lossy quantization, because the memory it frees up converts directly into more concurrent requests or longer supported context at the point where compute actually happens. That's the tier where quantization pays for itself immediately, no offsetting cost anywhere else in the pipeline.

NVMe offload changes the shape of the problem entirely. LMCache moves KV caches out of GPU memory and shares them across engines and across queries, instead of pinning every cache to the GPU that produced it. The mechanics behind that gain include batched data movement instead of tensor-by-tensor transfers, pipelining compute against I/O, a KV cache connector built as a separate module decoupled from inference engine changes, and a control API that handles pinning, lookup, cleanup, movement, and compression across GPU, CPU, storage, and network layers.

Network transfer is its own bottleneck, and a bigger one than most teams assume until they actually measure it. KV cache transfer alone can account for up to 42.2 percent of total job completion time, which should reset how a team thinks about where its latency budget actually goes https://arxiv.org/pdf/2607.28150.

Eviction, selection, and compression as a unified system: where format-native strategies fit in the serving stack

None of this works as an isolated trick. Compression has to sit inside a system that also decides what to evict, what to keep, and what to prioritize, and those three decisions are coupled tightly enough that solving them separately produces a worse system than solving them together.

PagedEviction operates at block granularity, aligned with the same paged block pool PagedAttention already uses. That alignment isn't incidental: block-wise eviction stays compatible with per-block quantization scales, so a system can evict whole blocks and requantize what's kept without the bookkeeping falling apart. Entropy-guided caching pushes the same logic further, allocating cache budget based on how much attention entropy a given layer carries. The implication is straightforward once you sit with it: not every layer deserves the same precision. Layers where attention spreads across many tokens carry higher entropy and are worth keeping at BF16, while layers where attention concentrates narrowly can absorb aggressive compression at little accuracy cost.

Streaming LLM, built for infinite-length generation, keeps the first four attention sink tokens plus a recent sliding window and lets everything else go. It draws a hard line between what has to be preserved losslessly (the sink tokens and the recent window) and what's a candidate for quantization or outright eviction (everything older than the window).

Agentic workloads break the assumption most of this tooling was built around: a single linear conversation growing forward in time. Leyline addresses that by introducing policy-driven cache editing through declarative span-and-replacement directives, letting a system rewrite sections of the cache instead of only ever appending to it. Leyline's splice kernel lifts replay cache-hit rate by 11.2 percentage points and cuts latency by up to 241 milliseconds https://arxiv.org/pdf/2606.01065.

The fragmentation numbers make the case for treating them as one problem rather bluntly. LLM inference systems waste 60 to 80 percent of allocated KV cache memory through fragmentation and over-allocation, and vLLM's PagedAttention reduces that waste to under 4 percent while enabling a 2 to 4× throughput improvement https://arxiv.org/pdf/2607.08057 https://introl.com/blog/kv-cache-optimization-memory-efficiency-production-llms-guide. AnchorKV pushes further still, shrinking the KV cache by 20× without discarding a single token, and SmartGen cuts time-to-second-token by up to 4.3× against a full KV cache transfer, while LMCache paired with vLLM reaches up to 15× throughput improvement on workloads like multi-round question answering and document analysis https://arxiv.org/pdf/2607.28150 https://arxiv.org/pdf/2510.09665 https://arxiv.org/pdf/2608.02901. None of these numbers come from the same lever. That's the point: format-native compression, smarter allocation, and workload-aware transfer each solve a different piece of the same budget, and a serving stack that only invests in one of them is leaving the other two on the table. Llama 3.1 70B at BF16 with 128K context and one concurrent user requires approximately 42.9 GB of KV cache memory https://arxiv.org/pdf/2607.08057. Llama 3.1 70B at FP8 with 128K context and one concurrent user requires approximately 21 GB of KV cache memory https://arxiv.org/pdf/2607.08057. Compared to FP16, the number of KV blocks for INT4 can be increased by 4× https://arxiv.org/pdf/2607.08057. Compared to FP16, the number of KV blocks for INT8 can be increased by 2× https://arxiv.org/pdf/2607.08057. Up to 3.4× KV cache capacity is provided by TurboQuant 4bit-nc https://vllm.ai/blog/2026-05-11-turboquant. TurboQuant 4bit-nc offers modest accuracy degradation of 1–4 points on most benchmarks https://vllm.ai/blog/2026-05-11-turboquant. TurboQuant k8v4 provides modest KV-cache savings of 2.4× versus 2× for FP8 https://vllm.ai/blog/2026-05-11-turboquant. KVmix-k2.28v2.56 maintains a comparable memory compression ratio of 4.8× https://arxiv.org/pdf/2506.08018. KVmix-k2.28v2.56 achieves superior inference acceleration of 5.23× https://arxiv.org/pdf/2506.08018. Transferring long-context prompts of 48K tokens takes 6.5× more time than the prefill computation https://arxiv.org/pdf/2607.28150. A 70B model with 8K context requires approximately 20GB cache per request https://introl.com/blog/kv-cache-optimization-memory-efficiency-production-llms-guide.