Networking Cost in Distributed LLM Inference Clusters

Networking cost in a distributed LLM cluster doesn't scale with GPU count or how often you push out fresh model weights. It scales with how much KV cache data moves between prefill and decode nodes, request by request, all day long. That single fact explains the cost structure of these clusters, and it explains why so much engineering effort now goes into transport layers, memory tiers, and cross-datacenter KV routing. Get the KV cache math wrong and every other optimization on top of it is wasted effort.

Start with the arithmetic, because it explains everything downstream. KV cache memory equals 2 times layers times KV heads times head dimension times sequence length times batch size times bytes per element. The 2 is there because the system stores both the key tensor and the value tensor for every token, at every layer, for every attention head. Cache size grows linearly with context length and with batch size, so serving more concurrent long-context users multiplies cost in a straight line. There's no curve that bends back down to save you.

Teams moving to long-context serving for the first time get tripped up by one thing: KV cache size is driven almost entirely by KV head count and layer depth, not by total parameter count. A model with fewer parameters but deep layers and many KV heads can carry a heavier cache burden than a bigger model built differently. Take a large model in the 70-billion-parameter class at a 128K-token context: the KV cache runs around 40 GB, while the model weights themselves take up roughly 140 GB in FP16. The cache isn't the biggest number on the page, but it's the one that grows every time a new long-context session opens, and that's what makes it the operational headache, not the weights everyone stares at first.

Monolithic prefill-decode colocation's GPU waste and the pressure to disaggregate

Prefill and decode are not the same job wearing two hats, and treating them as one is where most of the wasted spend comes from. Prefill is compute-bound: it runs dense matrix multiplication across the whole input prompt at once, and tensor cores climb toward saturation, often hitting somewhere around 92% utilization during a prefill burst. Decode works the opposite way. It's memory-bound, reading KV tensors for every token generated so far, one step at a time, and it lives or dies on HBM bandwidth rather than raw compute throughput.

Putting both phases on the same GPU produces a strange utilization pattern. Tensor cores spike near saturation during prefill, then ten milliseconds later, during decode, utilization on those same chips drops to around 28%. The hardware sat there, fully paid for, mostly idle, for much of the request's lifetime. Multiplying that pattern across a fleet running thousands of concurrent sessions turns the waste into the line item finance actually asks about.

LLM inference has stopped being purely a compute problem. It's turned into something closer to a content-distribution problem, where the "content" is KV cache state and the job is getting it to the right place at the right time without stalling a user's request. Systems research through 2025 and into 2026 lands on a blunt conclusion: monolithic colocation doesn't work for any workload that needs sub-20ms P99 decode latency and sub-200ms time-to-first-token under heavy concurrent traffic. The two phases want different hardware behavior, and asking one GPU pool to serve both well, at scale, just doesn't hold up. Anyone still running colocated prefill and decode at that latency bar is paying for GPU cycles that spend a substantial share of each request sitting underutilized.

What prefill-decode disaggregation requires the network to carry

Splitting prefill and decode onto separate GPU pools fixes the utilization mismatch, but it creates a new problem: every finished prefill produces a KV tensor that has to travel from the prefill GPUs to the decode GPUs before the model can generate a single output token. There's no way around this transfer. It sits directly on the critical path of every request.

It isn't a small side payload tucked quietly into the pipeline, either. At 128K context, a single request's KV state runs around 40 GB. Scaling that to a batch of 32 requests at a more modest 8K context each requires roughly 640 GB of KV cache memory. That's the data volume the network has to move, batch after batch, continuously, for as long as the cluster is serving traffic.

The TTFT overhead from KV transfer raises latency numbers directly. Measured deployments put KV transfer overhead at roughly double the raw prefill time, added straight onto time-to-first-token. That's a direct, measurable cost sitting between a finished prefill and the first token a user sees, not a rounding error. It's a direct, measurable cost sitting between a finished prefill and the first token a user sees.

This differs from other network traffic patterns inference engineers already know well. Model weight traffic loads once at startup and stays largely static after that. Training gradient traffic follows all-reduce collective patterns, synchronized across a job's lifetime. Disaggregated inference traffic looks like neither: it's a continuous, per-request stream of large tensor transfers, arriving unevenly, each one gating the next stage of a live user request.

The transport layer: NIXL, RDMA, and what distinguishes KV transfer from generic tensor communication

By 2026, NIXL, NVIDIA's Inference Xfer Library, has become one of the main mechanisms handling this cross-node KV transfer, showing up in both vLLM and NVIDIA Dynamo. It moves KV tensors between nodes over RDMA or TCP, picking whichever path fits the deployment.

RDMA matters here for a specific reason: it lets data move GPU to storage or GPU to GPU without routing through the traditional kernel and CPU-mediated network stack, cutting latency and lifting throughput in the process. Vendors building around exactly this principle have begun integrating RDMA support directly into their AI data platform offerings.

Inside vLLM, the kv_connector parameter picks the connector implementation, options like MooncakeConnector or NixlConnector, and that choice decides which underlying transfer mechanism handles the job: RDMA, NCCL, or plain TCP. RDMA paths win out for the large, latency-sensitive tensors KV transfer produces, where every millisecond shaved off the transfer reduces time-to-first-token directly. Picking TCP for this workload leaves latency on the table for no good reason.

KV transfer is a different animal from the collective communication patterns training engineers are used to. It's point-to-point, moving from one prefill node to one specific decode node. It fires per-request, not per-step. And it has to finish completely before decode can start, which makes it behave more like a synchronous RPC call than the pipelined, overlapped gradient exchange that defines training all-reduce. Anyone coming from a training background and expecting the same tolerances will misjudge the design constraints here.

Prefix caching and context reuse's effect on the networking calculus

Not every prefill needs to happen from scratch, and that fact changes the whole cost picture. LLM workloads circle back over overlapping context constantly: chat sessions share the same long system prompt across turns, RAG pipelines reuse retrieved passages across many queries, and agentic workflows branch out from shared intermediate reasoning states. Previously computed KV state, in each of these cases, can serve more than one request.

KV cache storage and reuse is the single most effective lever for cutting inference cost and latency, more effective, request for request, than most compute-side optimizations, and teams that skip straight to buying more GPUs before fixing cache reuse are solving the wrong problem. PagedAttention did a lot of the groundwork here, cutting memory waste from a range of 60-80% down to under 4%, which meant systems could afford to keep far more KV blocks resident instead of evicting them the moment memory got tight. That alone produced throughput gains of several times over across serving deployments.

LMCache, the open-source project now backing vLLM, SGLang, and NVIDIA Dynamo, builds directly on that opening. By persisting KV blocks across GPU memory, DRAM, disk, and object storage, it reports significant latency reductions on workloads with heavy cache-hit rates. Paired with vLLM specifically, the combination reaches a dramatically higher throughput improvement on multi-round conversation and document-analysis workloads, exactly the tasks where the same context gets revisited over and over inside one session.

Memory tiering as a network load-shaping tool: DRAM, NVMe, and pooled storage

Reuse only helps if there's somewhere to keep the cache between hits, and that's where the memory hierarchy comes in. Production systems now run a four-tier stack: GPU HBM holds the active decode batch, fastest and scarcest by far; CPU DRAM serves as swap space, a pattern already supported directly in LMCache and vLLM; local NVMe SSD picks up the overflow for long-context sessions or blocks evicted from DRAM; and networked or distributed storage pools handle cross-node reuse, letting one node's cached context serve requests landing on another node.

NVMe changes the economics of long-context serving in a specific way. Once a single long-context session's KV state crosses somewhere around 300-350 KB per token position at scale, it exceeds what a GPU's HBM can hold on its own. Offloading that overflow to NVMe lets a single H100 serve a meaningfully larger number of concurrent users than HBM capacity alone would allow.

That pattern already appears in hardware procurement. Node specifications going into 2026 increasingly call for high-capacity RAM and high-endurance NVMe drives sized specifically for KV offload duty, because this traffic pattern is continuous and write-heavy in a way that stresses drive endurance differently than typical storage workloads. Buying the wrong grade of SSD for this job turns endurance into a real operational failure, not a theoretical one, and teams that spec commodity drives here end up replacing them mid-cycle.

NVIDIA's cache management platform standardizes this offload path, pushing KV cache out to NVMe SSDs through a four-tier hierarchy and making that NVMe-resident cache part of the addressable context memory space rather than treating it as ephemeral working memory. It's backed by the BlueField-4 DPU, which offloads data-path operations to dedicated hardware rather than leaving them to the host CPU.

What disaggregation costs in practice: the GPU fleet economics

None of this comes free, and the size of the tax deserves honest accounting before totaling up the savings. That TTFT overhead from KV transfer, roughly double the raw prefill time, is a real cost sitting on top of every disaggregated request, and any argument for disaggregation has to clear that bar before it's worth adopting.

It clears the bar by a wide margin, and the fleet numbers make the case better than any theoretical argument could. Simulated fleet scenarios show disaggregation cutting annual GPU cluster cost by 35-46% against an all-aggregated setup running the same workload. One configuration puts this in dollar terms directly: an H100P plus H100D setup, 4 GPUs total, split one prefill and three decode, runs around $141,000 a year, against roughly $211,000 a year for an all-H100 aggregated configuration using 6 GPUs, both measured at the same Azure arrival rate and the same service-level objective.

Throughput numbers back the same story at larger scale. SGLang's disaggregated serving of DeepSeek-R1 across 96 H100 GPUs, split 3 nodes and 24 GPUs for prefill against 9 nodes and 72 GPUs for decode, delivers 52,300 input tokens per second and 22,300 output tokens per second. Moving the same workload to GB200 NVL72 hardware widens the gains further, with prefill throughput and decode throughput both climbing well past the H100 baseline.

The TTFT tax is a fixed toll that buys a considerably larger, variable return in GPU utilization and total fleet spend. It's a fixed toll that buys a considerably larger, variable return in GPU utilization and total fleet spend. Pay the toll once per request, recover it many times over in hardware efficiency across the fleet, and the math stops being close.

Agentic workloads and the emerging cross-datacenter KV transfer problem

Agentic systems push all of this further than chat or single-shot inference ever did. Context accumulates across a multi-step agent loop, and by turn 30, input token counts can run around 10 times what they were at turn 1. KV state generated during early reasoning steps has to survive across tool calls, code execution steps, and re-entries into the model, often on different nodes than where that state was first computed.

KV state in agentic workflows can't live comfortably inside a single linear decode stream the way it might in simpler serving patterns. It has to cross at least one boundary, whether that's between sessions, between nodes, or between memory tiers, and whoever designs the serving system has to define, explicitly, what each of those crossings costs and how it behaves. Skip that step and latency rises as an unexplained tax nobody budgeted for, visible once the bill arrives.

A recent paper by Ray, Feamster, and Jiang (arXiv:2608.01526) frames this as something close to an "Internet for KV Cache." The argument: once storing and reloading KV state gets cheaper than recomputing it from scratch, the best placement for compute and storage no longer stays confined to a single datacenter. It crosses cloud and geographic boundaries instead. At that point the network becomes an active distribution channel rather than background plumbing, and bandwidth, latency, and pricing start shaping KV management policy the same way content-delivery economics shaped where video got cached and served two decades ago.

The paper puts a number on the tradeoff. Transferring a 1 GB KV cache, roughly 100,000 tokens under DeepSeek-V4-Pro's compressed format, over a typical wide-area link carries a non-trivial cost under standard cloud egress pricing. Setting that against the cost of recomputing that same cache on an a high-end cloud GPU instance turns the comparison into a real economic decision rather than an engineering curiosity, one that has to be made fresh for every workload, every context length, and every pricing tier a provider happens to be sitting on that week.