I/O Bottlenecks That Stall GPU Utilization in LLM Serving

LLM serving stalls when the I/O hierarchy cannot supply KV cache data fast enough to keep GPUs busy. The remedy is to identify the tier that limits data movement, then address its bandwidth, latency, control overhead or scheduling constraints.

Why GPU utilization collapses during LLM serving — and why compute is not the cause

A GPU can have ample compute capacity and still spend much of a serving request waiting. Understanding Bottlenecks for Efficiently Serving LLM Inference With KV Offloading found that GPUs serving KV-offloaded requests consumed only 28% of their rated TDP. That power measurement supports the study’s finding that data transfer dominates execution.

The mechanism is the KV cache, which retains key and value vectors from previously processed tokens so generation can reuse them. This exchanges computation for memory access. When the cache no longer fits in GPU memory, that exchange depends on the slower tiers below it. The resulting stall is a supply problem.

How KV cache size scales with context length, batch size, and model architecture

The cache grows with both the history each request retains and the number of requests served together. Its size follows the formula KV bytes = 2 × layers × KV heads × head dimension × sequence length × batch size × bytes per element. Each factor multiplies the others, so concurrency magnifies the pressure from longer contexts. Under long-context concurrent load, KV storage can exceed the model weights themselves.

A long-context workload with eight concurrent users can demand roughly 320 GB of KV storage, four times the H100's total HBM. Context windows reaching hundreds of thousands or even millions of tokens make this a production capacity constraint. Retaining the state for multiple users therefore forces cache blocks into slower memory or storage.

Grouped Query Attention (GQA), as used in Llama 3, reduces KV heads relative to query heads, while MLA attention reduces data demand further. For a single long-context Llama 3 70B request, FP8 reduces the BF16 footprint to roughly half and NVFP4 to roughly a quarter. Those gains delay offload, but they do not eliminate it at scale.

The memory hierarchy's bandwidth and latency gaps at each tier

Once cache blocks leave HBM, where they are stored sets how quickly they can return. The hot GPU HBM tier supplies roughly 3.35 TB/s with sub-microsecond latency. Warm CPU DRAM reached over PCIe 5.0 x16 supplies roughly 63 GB/s at 10–100 µs. That path offers more than 50× less bandwidth than HBM, so available host memory is not equivalent to available GPU memory.

The cold NVMe SSD tier over PCIe 4.0 falls further, to roughly 7 GB/s and 100 µs to 1 ms. Its bandwidth is nearly 500× lower than HBM. These slowdowns have different causes: bandwidth limits how much data can keep moving, while latency adds to the wait for the blocks needed. A large cache restoration exposes the bandwidth gap, while fragmented requests also expose the cost of individual I/O operations.

CXL 1.1 memory expanders provide bandwidth between NVMe and HBM, but remain far below HBM. Multi-GPU relay can increase theoretical host-read bandwidth through peer PCIe links. However, that improvement can move the binding constraint to host DRAM bandwidth or inter-socket interconnects. The important unit of analysis is consequently the complete route to the GPU.

Why the PCIe boundary is the most consequential single bottleneck

PCIe is often the first boundary where cache reuse becomes more expensive than the compute it was meant to save. Understanding Bottlenecks for Efficiently Serving LLM Inference With KV Offloading defines κcrit as the cached-to-prefill token ratio at which execution becomes memory-bound. On real hardware, the threshold is often well below 100, while the ratios for document-query workloads reach into the thousands. The paper found that transfers consumed 99% of latency in KV-offloaded serving.

GPU Direct Storage (GDS) removes the CPU from the NVMe-to-GPU data path, but Tutti documents that the CPU remains on the critical I/O control path. Fragmented GPU memory produces many tiny random I/Os that require CPU initiation. Under GDS, CPU overhead scales as O(layers × blocks), coupling control cost to both model depth and cache fragmentation.

Tutti’s motivating vLLM regression shows why that distinction matters. NVMe-backed KV restoration that had helped earlier vLLM versions became harmful as compute efficiency improved. The compute path became faster, but CPU-mediated restoration consumed a larger share of total latency.

How KV cache restoration becomes a batched scheduling problem, not a per-request one

Even when the transfer path is understood, choosing whether to load or recompute each request’s cache on its own is not enough. Recomputation grows quadratically with sequence length, while I/O grows linearly but remains bandwidth-constrained. In a shared batch, those choices compete for the same compute and I/O resources. One slow restoration can become a straggler that delays the batch’s decode phase.

CacheFlow from UIUC and NUS treats restoration as a multi-dimensional scheduling problem. It overlaps recomputation of earlier tokens with transfer of later tokens, using causal dependencies. It also pipelines lower-layer recomputation with upper-layer KV transfer and lets GPUs reconstruct their own model-shard caches concurrently.

CacheFlow’s batch-aware two-pointer scheduler prioritizes operations by their marginal reduction in recomputation cost. The CacheFlow paper reports 88% GPU utilization together with high I/O utilization and substantially reduced time to first token (TTFT). The scheduling insight is broader than the algorithm. Scheduling must aim for overlap across the batch, because a restoration choice that works well for one request can still waste shared resources.

Moving the CPU off the critical I/O path with GPU-centric KV cache management

Batch-aware scheduling addresses contention, but CPU-mediated I/O can still prevent the system from issuing enough work. Tutti responds with a GPU-centric two-tier HBM-SSD KV cache object store. The CPU loads I/O kernels asynchronously once per layer. This reduces CPU overhead from O(layers × blocks) to O(layers).

With the CPU off that critical path, the GPU can issue large parallel I/O requests for KV cache objects directly to SSDs. That exposes parallelism which the CPU-mediated GDS control path cannot exploit.

This is an architectural response rather than a faster version of the same control loop. It does not make SSD bandwidth equal to HBM bandwidth. Instead, it prevents CPU orchestration from becoming an additional restriction on the storage path. The distinction matters whenever a nominally direct data path still depends on host intervention to make progress.

RDMA as the high-throughput path for disaggregated KV cache transfer

Disaggregated inference introduces another boundary by separating prefill workers from decode workers. The cache produced during prefill must reach the decode worker before generation begins. Transfer latency therefore becomes TTFT overhead, making KV transport a prerequisite for useful disaggregation. A fast worker cannot compensate for state that has not arrived.

RDMA provides the high-throughput path through kernel-bypassing data plane operations. However, NIC-specific implementations have made that path difficult to adopt across hardware environments. TransferEngine / fabric-lib supplies a portable RDMA abstraction and demonstrates peak throughput on both NVIDIA ConnectX-7 and AWS EFA. Portability matters because an inference engine needs a usable transport across its deployment hardware.

NVIDIA Dynamo uses NIXL (NVIDIA Inference Xfer Library) to transfer KV cache between prefill and decode workers. LMCache shares the same NIXL primitives. These components give serving stacks RDMA-backed transport, so each engine can use it without building its own hardware-specific path. This helps at the network boundary, where disaggregation would otherwise ease local memory pressure at the cost of more transfer delay.

Lossless compression as a bandwidth multiplier for BF16 and FP8 KV data

Transport improvements increase the rate at which bytes move, while lossless compression reduces the bytes required. BF16 and FP8 tensors have low-entropy exponent distributions that make this approach useful. For BF16, skewed exponent values carry less information than the bits set aside for them can hold. Compression can exploit that structure without discarding precision.

Huff-LLM reports 17–33% compression for FP16 and BF16 models, with higher compression ratios for BF16 because of its larger number of exponent bits. Those model results establish the value of format-aware lossless methods, while SplitZip targets KV compression specifically for disaggregated serving. Smaller KV blocks let each unit of PCIe, NVMe or network bandwidth carry more useful cache data.

This differs fundamentally from quantization. FP8 and INT4 reduce footprint through lossy precision reduction, whereas lossless compression preserves the data being compressed. The techniques are therefore complementary. For agentic workloads where output errors compound across steps and full precision is required, lossless compression offers a bandwidth lever without imposing another precision tradeoff.

Hardware-level responses — ICMSP, NVLink C2C, and unified HBM approaches

ICMSP uses BlueField-4 DPUs to address the NVMe tier by offloading PCIe I/O control, keeping GPU streaming multiprocessors focused on the forward pass. Its internal KV transport is managed by NVIDIA DOCA Memos on those DPUs. NIXL is the transfer library inference frameworks use to interface with the platform.

That separation connects ICMSP to disaggregated inference without making the platform and transport library interchangeable. Dynamo uses NIXL for prefill-to-decode KV transfer, while the DPU-based platform addresses storage-side movement. ICMSP’s applicability depends on both the binding tier and the availability of the required BlueField-4 DPUs in the server, whose availability in cloud environments today is limited.

NVLink C2C and unified HBM approaches target a different part of the problem. The KV offloading bottleneck analysis identifies them as ways to raise κcrit substantially, allowing higher cached-to-prefill token ratios before execution becomes memory-bound. MoE-Lens also shows why model architecture can complicate this picture, since large expert parameter sets create memory pressure and CPU-GPU resource interactions. Those interactions warrant separate analysis rather than an assumption that every serving bottleneck is KV transfer alone.

A diagnostic framework for locating the actual binding constraint in a given deployment

Diagnosis begins with the workload’s cached-to-prefill token ratio relative to κcrit. If the ratio is far above the system’s threshold, more compute leaves the limit on the transfer path in place. If HBM is exhausted while DRAM remains available, the first branch concerns capacity efficiency, including PagedAttention fragmentation and KV quantization. If those levers are already applied, DRAM offload is the next option.

Once offload is necessary, the next branch is the constrained boundary. If PCIe is binding, auditing CPU involvement in GDS distinguishes control overhead from the link’s bandwidth limit. GPU-centric I/O or ICMSP DPU offload addresses the control architecture. If NVMe throughput is limiting instead, compressing KV blocks before SSD writes increases effective bandwidth, while multiple NVMe devices in parallel provide the hardware path. If the network is limiting disaggregated inference, kernel-bypassing RDMA and portable implementations such as TransferEngine/NIXL address that boundary.

These choices are calibrated to long-context, multi-user, batched serving. Short-context single-user workloads do not gain useful capacity from NVMe offload and instead incur extra I/O latency. Even within the intended workloads, a successful intervention can expose the next constraint in the hierarchy. As inference engines improve compute efficiency, unchanged I/O overhead occupies a larger share of latency. Storage and data movement therefore require ongoing engineering discipline.