Latency Throughput and IO Contention the Serving Bottlenecks That GPU Benchmarks Never Show

Long-context LLM serving infrastructure requires more than high-throughput GPUs. At 128K contexts and beyond, serving speed, economy, and feasibility depend on KV cache allocation, memory-tier placement, and data movement. GPU arithmetic benchmarks miss these constraints, so infrastructure decisions must rest on cache reuse, mixed-load latency, storage I/O, and transfer measurements across the actual interconnect.

Why GPU benchmarks are designed for a problem that production serving doesn't have

FLOP benchmarks have an intuitive appeal because they are reproducible, comparable, and convenient for hardware vendors. They describe how well a GPU performs arithmetic at saturation. Production serving asks a different question entirely. Its performance depends on whether data reaches compute units quickly enough under changing request lengths, cache occupancy, and concurrent load.

Fixed, synthetic request lengths conceal those dynamics. Dell Technologies’ KV Cache Optimization Strategies for Scalable and Efficient LLM Inference, published March 24, 2026, describes how KV cache memory grows linearly with context length, constraining memory capacity, bandwidth, and inference throughput. A FLOP count captures none of that growth. Teams that procure and configure hardware against arithmetic throughput alone therefore optimize for a regime that does not describe their serving workload.

What LLM serving actually does during inference, and why it looks nothing like a benchmark run

Inference has two phases with different resource profiles. Prefill processes the prompt and can saturate compute, while decode generates one token at a time and is memory-bandwidth-bound. Decode achieves only a small fraction of peak model FLOPs utilization, far below prefill efficiency. During this phase, the GPU spends much of its time waiting for data movement. For most serving workloads, decode dominates wall-clock time, so a single throughput figure obscures the phase that occupies the system longest.

The memory hierarchy compounds this structural asymmetry further. It spans six tiers, from registers down to NVMe, with orders of magnitude separating the fastest and slowest. Model weights and KV caches both occupy HBM during serving, competing for the bandwidth needed to feed inference. When a system waits on a slower tier, it takes longer to generate tokens. Compute benchmarks that assume data is already staged do not reveal the resulting bandwidth constraint.

How KV cache fragmentation silently caps throughput before any request hits the GPU

Pre-allocation strategies reserve space for worst-case sequence lengths even though most requests never fill the maximum context window. The resulting fragmentation and over-allocation can dominate wasted memory in production serving. Capacity is reserved but does no useful work. That reservation caps how many requests the system can take on before arithmetic throughput becomes a limit.

For a large model with a long context window, the KV cache alone can reach tens of gigabytes per request. Across a large batch, it can take up more memory than model weights. Fixed-length benchmarks conceal this problem because their requests do not reproduce the distribution of real sequence lengths. They can therefore make memory that exists on paper look like memory the system can use to serve requests.

vLLM provides a concrete example of serving infrastructure that treats KV cache management as a throughput concern. Fitting more requests into existing memory can improve throughput without faster arithmetic.

How prefill and decode interfere when co-located, and what disaggregation reveals about the real cost

Co-locating prefill and decode introduces contention between their different resource demands. Although prefill is compute-intensive and decode is bandwidth-intensive, both use the same HBM bandwidth and compute units. Under realistic load, interference can multiply per-token latency and inflate p99. A system can have high throughput overall while some requests face delays.

SGLang ran a 96-H100 DeepSeek-R1 deployment with 3 prefill nodes and 9 decode nodes. That 1:3 split makes phase allocation an explicit engineering variable, one absent from a single-GPU benchmark. The appropriate ratio depends on the workload. Short inputs with long outputs demand a different balance from agentic traffic with very long prompts and tiny outputs.

KV tensors produced during prefill must reach the decode node, and a long-context request on a large model can require tens of gigabytes of transfer. If that movement takes longer than decode itself, separation degrades latency.

Why interconnect topology determines whether disaggregated serving works or backfires

The cost of that transfer depends on physical placement. Bandwidth between GPUs varies by orders of magnitude depending on how they connect, so the same serving software can behave differently on different interconnect paths. A transfer acceptable over NVLink may be unacceptable across nodes over InfiniBand. The relevant quantity is the time required to move the request’s KV cache between its assigned prefill and decode resources.

For a large model with a long context, cross-node transfer alone can exceed the time decode would have taken. That makes topology-aware KV routing a prerequisite for realizing disaggregation’s potential gains. A benchmark confined to one GPU cannot reveal this dependence. Even separate compute and bandwidth results leave the central comparison unresolved unless transfer latency is measured against decode time on the deployed path.

What infrastructure is needed when context windows reach hundreds of thousands of tokens?

The Dell survey documents context windows expanding from tens of thousands to hundreds of thousands, and in some cases millions, of tokens. At these lengths, a single request’s KV cache can exceed single-GPU HBM capacity. Serving then becomes a storage problem as well. NVMe offload is no longer an exceptional provision for an oversized request, but an operating condition the infrastructure must accommodate.

NVIDIA announced its Inference Context Memory Storage Platform at CES 2026 to standardize KV cache offload across GPU HBM, CPU DRAM, and NVMe SSD. BlueField-4 DPUs handle I/O scheduling to prevent GPU stalls on storage access, targeting up to 5× token throughput improvement. The announcement signals that offload has become a production infrastructure concern. It also puts memory placement and I/O scheduling directly in the performance path.

Multi-turn sessions expose a further distinction between server-internal NVMe and remote shared storage when context must persist across requests or move between serving nodes. GDS reduces CPU overhead by routing NVMe reads directly into GPU HBM through DMA, but its benefit centers on large sequential reads. Random-access, small-block KV workloads remain largely unaddressed by that optimization alone. MLPerf Storage v3.0, released September 2026, adds a KV cache benchmark to test this problem.

What agentic workload traces reveal about the metrics that actually determine user experience

Production Claude Code sessions show how far real traffic can differ from what benchmarks assume. The median main-agent request had a very large input and a small output, while the great majority of main-agent turns reused at least 90% of their input. For these workloads, prefill cost and time-to-first-token, or TTFT, govern latency and cost more directly than the rate of output-token generation.

A realistic agentic trace evaluation raised cache hit rate from negligible levels to 92.2% through a distributed KV store. Throughput, P50 TTFT, and end-to-end latency all improved substantially through cache management on existing silicon. A system that repeatedly recomputes reusable context spends resources on work the workload did not require.

Leyline: KV Cache Directives for Agentic Inference identifies another mismatch between conventional caching and agentic traffic. Exact-prefix caching assumes prompts grow monotonically, but agents retry failed tool calls, discard stale outputs, and pivot their trajectories. Such edits can invalidate prefix matches even when the underlying KV tensors remain valid. Leyline introduces a declarative (span, replacement) 4-tuple directive for explicit cache edits without re-prefilling everything downstream. On debug-gym, using that interface for a ten-line truncation rule improved solve rate by more than fourteen percentage points, which shows that cache architecture can affect how well tasks are done as well as latency.

How compression at the KV cache level changes the storage and bandwidth calculus

Moving a large model’s weights from BF16 to FP8 can substantially reduce their footprint and potentially reduce the number of accelerators required. Yet the same model’s KV cache at a 128K context can exceed 100 GB independently of weight precision. The cache needs its own infrastructure decision. Using model size as the full memory budget overlooks how memory needs grow with context and concurrency.

KV cache compression also changes the transfer and storage constraints established earlier. A smaller cache reduces the data that must cross the prefill-to-decode interconnect and occupy offload tiers. Its benefits therefore extend beyond fitting requests into HBM. AI-native formats such as BF16 and FP8 need more than general-purpose compression can offer. The Dell survey finds that no single technique works best across deployment scenarios and points instead to adaptive, multi-stage pipelines that match the type of workload.

What MLPerf has started to measure and what it still cannot see

MLPerf Inference v6.0, published April 2026, added GPT-OSS 120B, a 120B-parameter text-generation model, among several new benchmarks. The previous GPT-J 6B task had been retired because all major accelerators had saturated it. Benchmark workloads themselves need to remain discriminating tools. Larger model tests address that problem, but they do not by themselves expose storage contention or cache reuse.

Storage v3.0 adds KV cache and vector database workloads and opens the suite to S3 object storage. Its KV test simulates multi-turn conversations that write context once and read it repeatedly, using transfers from 64 MiB to about 3 GiB and a median workload totaling 13 TiB. MLCommons’ Brian Belgodere described the purpose explicitly. “Including tests that decompose monolithic AI systems and focus on specific storage uses and patterns — such as checkpointing, KV caching and vector databases — gives stakeholders a much clearer idea of how to engineer and provision AI systems to minimize storage performance bottlenecks”.

MLPerf does not yet measure the compound effect of cache hit rate variance, storage I/O latency jitter, and interconnect topology under simultaneous mixed prefill/decode load. Anyscale’s Ray Serve reported dramatically lower latency and higher throughput after upgrades to its online inference path in March 2026. Existing benchmarks cannot decompose the contributions of batching, caching, and scheduling behind such a production result.

What engineers building production serving infrastructure should actually measure

For agentic and multi-turn traffic, teams should measure cache hit rate along with TTFT on requests like those the system will serve. Per-request allocation tracing should record actual page utilization, exposing reserved capacity that theoretical memory figures hide.

Prefill/decode interference should be measured at p99 under mixed load. Tail latency exposes contention before averages do. For disaggregated deployments, KV transfer time should be compared with decode time across the actual interconnect path. This comparison establishes whether phase separation removes more delay than it introduces.

NVMe-offloaded KV cache requires storage testing under concurrent request load, using the random-access, small-block patterns of serving alongside sequential bandwidth measurements. The benchmark suite should combine MLPerf Inference load profiles, Storage v3.0 KV workloads, and topology-aware transfer measurements. Memory tier, offload strategy, compression format, and interconnect configuration must then be evaluated against the context lengths and concurrency that production traffic demands, because those choices constrain the useful work the available compute can perform.