Prefill vs. Decode GPU Utilization Imbalance

Prefill and decode GPU utilization imbalance is a hardware problem before it is a scheduling problem. Prefill consumes compute capacity, while decode depends on memory bandwidth. That asymmetry determines when chunked prefill is sufficient, when separate GPU pools pay off, and when cache movement becomes the limiting cost. Architectural choices succeed only when they address the resource the workload actually exhausts.

Why prefill and decode make fundamentally different demands on GPU hardware

Prefill processes the entire input sequence in a single forward pass, using large matrix-matrix multiplications that can saturate tensor cores. Compute utilization rises sharply, with power draw at 70–100% of TDP. Decode has a different resource profile. Its autoregressive loop is memory-bound, so available memory bandwidth constrains progress while much of the GPU’s compute capacity remains unused. Decode consequently draws substantially less power than prefill.

Prefill determines the compute contribution to time-to-first-token, or TTFT, while the decode loop determines time-between-tokens, also called time-per-output-token, or TPOT. A GPU can be well suited to producing the first token and poorly utilized while producing the rest, because the two phases ask different things of the same hardware.

How the GPU generation curve makes the decode problem worse over time

Successive GPU generations have increased compute capacity faster than memory bandwidth, widening the mismatch. Microsoft’s Splitwise paper, published at ISCA 2024, measured a 3.43× compute increase from A100 to H100, compared with only a 1.64× increase in memory bandwidth. A faster GPU offers far more arithmetic capacity without a corresponding increase in the resource that limits decode.

An upgrade to colocated serving will not speed up both phases in line with the amount spent on hardware. The compute-bound share of the workload does not expand simply because more compute becomes available. Decode remains constrained by data movement, leaving more arithmetic capacity idle during generation. Hardware progress therefore strengthens the case for treating the phases differently, even before their interference on a shared GPU is considered.

What happens when both phases share the same GPU: the interference tax

A large prefill request can interrupt otherwise steady decode service when both phases share a GPU. A 32K-token document upload arriving mid-decode, for example, can inflate TPOT by 2–30× for concurrent requests. One upload can stall every concurrent chat response.

The conflict is structural, although scheduling controls its duration. Prefill saturates compute units that decode does not need to saturate, but a waiting decode batch cannot reclaim those resources mid-batch. It waits for prefill work to yield. This connects hardware utilization directly to service-level objectives, because acceptable TTFT for the incoming document can coincide with unacceptable TPOT for existing users. The next constraint is whether the KV cache adds a memory-volume problem to that timing conflict.

How the KV cache turns the memory-bandwidth constraint into a capacity crisis

The KV cache makes autoregressive generation efficient, but its footprint grows with context length and concurrency. Its per-token size follows 2 × num_layers × num_kv_heads × head_size × bytes_per_element. The total size is that figure multiplied by batch_size × sequence_length. Each factor is a capacity lever. At long context, one user’s cache can occupy tens of gigabytes, so even a small concurrent workload can require multiples of a single GPU’s HBM capacity.

Moving cache blocks to CPU DRAM frees up capacity but adds the cost of transferring them. Understanding Bottlenecks for Efficiently Serving LLM Inference With KV Offloading formalizes a critical cached-to-prefill token ratio, κcrit, beyond which execution becomes memory-bound. Conversational and document-query workloads exceed that threshold by orders of magnitude. The paper reports cases where PCIe transfers consume 99% of execution time, with GPUs drawing only a fraction of rated TDP. Prefix reuse can therefore turn otherwise compute-bound prefill into an off-GPU cache-loading stall, exposing the steep bandwidth drops between memory tiers.

The bandwidth cliff: why each tier of the memory hierarchy changes the problem

HBM supplies bandwidth in the TB/s range, while the PCIe 5.0 CPU-GPU interconnect provides roughly 2 percent of HBM’s bandwidth. CPU DRAM may hold the cache, but the interconnect governs how quickly the GPU can receive it. Capacity and access speed are separate constraints. With long contexts and heavy prefix reuse, cache transfer can dominate end-to-end latency so completely that faster GPU computation contributes little to the user’s experience.

NVMe SSD sits another bandwidth tier below CPU DRAM, but raw bandwidth alone gives an incomplete cost model. Loading reusable KV blocks from NVMe can still be substantially faster than recomputing a long, highly overlapping prefix. The comparison is against the work avoided. Offloading therefore has two distinct costs to evaluate, namely the time spent retrieving stored state and the compute time that retrieval replaces. Placement can produce a latency win even when it uses the slowest tier.

Chunked prefill: the scheduling fix that works without adding infrastructure

Chunked prefill breaks long prompts into smaller units interleaved with decode batches on the same GPU. Each chunk occupies compute resources for a bounded duration before yielding, so decode no longer waits through an entire long prefill. This works without a separate pool or a network transfer.

The Nexus intra-GPU disaggregation paper identifies the remaining limitation. Batching prefill chunks with decode improves utilization, but compute-heavy and memory-sensitive work still produces fine-grained interference within the shared batch. Chunking bounds the ongoing disruption. That distinction does not make it an inadequate solution, because bounded interference may be enough to stabilize TPOT under the workload’s latency requirements. For teams choosing an initial response, chunked prefill is the appropriate default below the scale where specialization gains outweigh integer GPU allocation losses and the overhead of managing separate pools.

Engine-level prefill-decode disaggregation: what it solves and what it costs

Engine-level disaggregation places prefill and decode on distinct GPU pools, removing their shared-GPU interference and allowing independent scaling and tuning. DistServe, presented at OSDI 2024, demonstrated more requests per GPU within the same latency constraints and substantially lower latency variance on separate A100 pools. Specialization also permits different hardware and power policies. Compute-rich GPUs suit prefill, while older or cheaper hardware with strong memory bandwidth can serve decode. AMD Research’s RAPID, published in January 2026, shows that role assignment combined with non-uniform power caps improves peak-load SLO attainment beyond static disaggregation.

Those gains have structural costs. Each completed prefill must transfer its cache, potentially multiple gigabytes per request, while small deployments lose utilization because fractional GPUs cannot be assigned to either pool. Scaling the pools separately also means tuning the prefill-to-decode, or P:D, ratio. Nexus offers a middle path, dynamically partitioning resources within one GPU and reporting throughput and latency matching or exceeding engine-level separation with half the GPU count.

Moving KV cache between GPU pools: the network layer becomes the bottleneck

Moonshot AI’s Mooncake, presented at FAST 2025, shows why disaggregation cannot be evaluated through GPU execution alone. In its separated deployment, prefill execution accounts for only a small fraction of P95 TTFT, while queuing and inter-node KV transfer account for the rest. At production scale, total KV transfer rates can exceed 100 GB/s, so the network hop can cost more than the interference it removes if the interconnect cannot keep up.

NVIDIA Inference Xfer Library, or NIXL, supplies a mechanism for moving cache tensors across these paths. Its supported paths include NVLink for single-node GPU transfers, InfiniBand RDMA and RoCE for multi-node movement, NVMe-oF for storage movement, and TCP as a fallback. Their bandwidth differs by orders of magnitude. Agentic and multi-turn workloads can also saturate prefill engines’ storage NICs while decode engines’ NICs remain idle, replacing compute imbalance with storage-network imbalance.

Quantization and compression as a lever on both the bandwidth and capacity problem

Reducing KV cache bit-width addresses capacity and movement together. Smaller entries allow more concurrent users to fit in HBM and reduce the volume crossing PCIe or the network. The benefit is independent of pool topology. TensorRT-LLM documentation reports that FP8 KV cache quantization enables substantially larger batch sizes on H100, directly addressing decode’s memory-capacity constraint.

The choice of format thus affects storage and bandwidth as well as model quality. BF16 and other AI-native formats need compression tools built for them because general-purpose tools do not use the statistical structure of transformer activations and weight distributions. KIVI, presented at ICML 2024, demonstrated plug-and-play 2-bit asymmetric KV cache quantization. Research into lossless and near-lossless approaches extends that direction as contexts grow. A commitment to FP8 KV storage changes the I/O profile through HBM, PCIe, and NVMe, while compression and decompression overhead must be considered at each boundary.

Formalizing the storage tier: NVMe and three-tier KV cache architecture

A three-tier cache architecture assigns distinct roles to GPU HBM, CPU DRAM, and NVMe SSD rather than filling each tier only after the preceding one overflows. Block placement, movement timing, and retrieval speed become scheduling decisions. Passive overflow is not a sufficient policy. At long context and high concurrency, the serving system remains useful only if these decisions allow retained state to be supplied quickly enough.

DPU-managed I/O can move the work of handling metadata and moving data off the GPU’s critical path. That separation addresses the management cost of a storage tier as well as its nominal bandwidth. NVMe sizing and eviction policy then determine how much reusable state remains available and which requests incur retrieval or recomputation costs. Since those choices directly affect long-context TTFT, storage has to be assessed for its role in inference execution as well as for the capacity it adds.

How benchmarks measure — and fail to measure — the prefill-decode tradeoff

MLPerf Inference specifies compliance thresholds for both TTFT and TPOT, but submissions report aggregate throughput. That aggregate does not decompose the phase tradeoff. Two systems can deliver the same throughput while using their latency allowances differently, with one permitting longer TTFT to preserve steady generation and the other doing the reverse. Both may satisfy compliance thresholds yet suit different workloads and user expectations.

The throughput figure hides more when production traffic comes in bursts and prompt lengths are heavy-tailed. A throughput figure alone does not show how long prefills disturb concurrent decode or how cache transfers consume the first-token budget. For infrastructure teams, the practical response is to instrument TTFT and TPOT separately under realistic workload distributions.

Reading the asymmetry: a decision framework for choosing among the architectural options

The choice depends first on model size, context length, what is in each batch, and cluster scale. Chunked prefill is the starting point when the primary symptom is long-prefill interference and separate pools would be underused. Pool separation pays under more specific conditions. Cluster scale must make integer allocation losses small, the workload must be prefill-heavy with a P:D ratio stable enough to tune, and a high-speed interconnect must keep KV transfers manageable.

Compression can be chosen separately because it cuts memory pressure and the amount of data to transfer under either topology. The architectural layers can then compose.

At long context with high prefix reuse, that composition depends on actively managed NVMe storage. How aggressively blocks are placed and evicted matters because retrieval policy sets the lower bound on long-context TTFT. Even a fast disaggregated cluster stays storage-bound when its storage path is slow. Storage is where the GPU-level design either gains access to the state it needs or waits for it.