Colocation vs. Hyperscaler Cloud for AI Inference Workloads

Colocation vs. hyperscaler cloud for AI inference is an engineering decision governed by sustained GPU utilization, KV cache residency, and storage throughput. Colocation makes sense when demand is steady enough to pay for dedicated infrastructure and the team can manage its memory and network hierarchy. Managed cloud wins when variable demand, geographic reach, or operational simplicity outweighs that control. Open-source LLM inference stack benchmarking and optimization must measure these conditions together.

Why the colocation vs. hyperscaler decision is an engineering problem

Production inference is latency-sensitive and often sustained, while training can be bursty and less tied to users’ locations. The distinction changes what infrastructure must deliver. Iron Mountain’s enterprise analysis places inference near users, with geographically distributed capacity. Within that deployment, memory bandwidth constrains token generation, so purchasing more compute does not necessarily resolve the bottleneck.

An hourly GPU price says little about the cost of serving a workload if the GPU spends much of its time idle or waiting for data. Sustained utilization connects the engineering design to the economics because it determines how much useful work absorbs the cost of installed capacity.

How sustained GPU utilization rate sets the economic floor for the entire decision

When sustained GPU utilization is above 55–60%, colocation or dedicated GPU infrastructure is a stronger choice over several years, if it also meets the team’s other technical and operational needs. Below that range, renting capacity can spare the team the cost of hardware it cannot keep busy.

That threshold is a screening criterion. A team at 65% utilization still needs to account for egress, storage I/O, and networking capex before choosing a deployment model. Teams in the middle range have the hardest choice because flexible rental and dedicated capacity each lack a clear edge over the other. For them, infrastructure below the GPU layer can determine which option is cheaper.

A workload with steady, high demand can spread the cost of owned hardware over the inference it keeps running. When demand varies, hardware sits idle between peaks, which weakens that advantage. This is why the evaluation must begin with the production workload. Only then does a comparison of infrastructure costs become meaningful.

The hidden costs that shift the breakeven: egress, storage I/O, and networking capex

A team running daily retrieval-augmented generation pipelines at scale needs to work out the cost of moving its data as well as renting its GPUs. Expected egress volume belongs in the cost model because a GPU-hour comparison excludes it. EBS io2 or FSx for Lustre supporting an active inference pipeline adds monthly cost beyond compute, with the required tier shaped by the model weights and KV cache footprint.

Disaggregated serving introduces a different cost on the colocation side. Dedicated high-speed networking takes capital expenditure to provide the bandwidth that the serving architecture needs. That spending buys control, but the team still has to count it in the comparison. A colocation facility with poor connections can also lose its compute advantage to latency because access to carriers and internet exchanges differs from site to site.

The practical evaluation is a full three-year TCO. It should include egress for the amount of data the team expects to move, storage tiers that fit the working set, and networking capex for what the serving design needs. These categories move differently between deployment models, so the cheapest GPU rate need not produce the cheapest inference service. The next constraint is whether that storage investment can keep reusable context available.

Why KV cache residency is the infrastructure constraint that makes or breaks colocation's advantage

The KV cache stores key and value vectors from previous tokens so generation avoids redundant computation. Its footprint grows linearly with context length. At long contexts, that footprint can exceed a single GPU’s HBM capacity, creating a residency problem. Hot blocks belong in GPU HBM for active requests, warm blocks can move to CPU DRAM, and cold blocks from paused or completed contexts need fast retrieval from NVMe.

LMCache connects this hierarchy to vLLM, SGLang, and NVIDIA Dynamo through configurable backends including local NVMe, Redis, and remote object stores. Reuse through an NVMe cache hit saves computation by skipping prefill for the cached portion of the prompt, while new tokens still require prefill. Its value depends on whether retrieval is fast enough for the serving workload.

Colocation lets operators specify drive count, PCIe generation, RAID configuration, and local or NVMe-oF storage. Managed instances abstract those underlying choices away. An emerging CXL-attached tier offers another topology, allowing GPU DMA to write KV blocks to shared memory that decode workers read without retransmission through a NIC. For contexts of 128K tokens and above, or high concurrency with substantial cache reuse, control over this hierarchy can be the mechanism that makes the cost and latency targets achievable.

How prefill/decode disaggregation changes the infrastructure requirements for production serving

Prefill and decode place different demands on hardware. Prefill is compute-bound because it processes the prompt, while decode is memory-bandwidth-bound because each token step reads model weights and the KV cache. Running both phases on identical GPUs creates interference that can reduce throughput. Separating them deals with that interference, but the infrastructure then needs to handle KV transfer between GPU pools.

At production scale, total cross-node KV transfer can exceed 100 GB/s, so the network needs RDMA-capable fabrics to keep it from becoming a bottleneck. Physical topology now shapes the serving design. NVLink operates within a node domain, InfiniBand or RoCEv2 connects nodes within a cluster, and TCP connects across data centers. Keeping disaggregated infrastructure in a physically controlled environment limits the topology spread that those transfers must cross.

NIXL supports over a dozen transfer backends, including UCX, Libfabric, GDS, POSIX, and S3-compatible object storage, so backend selection directly affects the latency floor. Mooncake, accepted at FAST 2025, makes the architectural trade explicit by using more storage to reduce computation in disaggregated serving centered on cached attention keys and values. Framework choice also affects reuse. Compared with vLLM, SGLang’s RadixAttention provides a throughput advantage on shared-context workloads through cached shared computation, but that advantage does not remove the need to provision the transfer path.

What the interconnect fabric decision means in colocation vs. managed cloud environments

The fabric decision concerns the complete KV transfer path. RDMA over InfiniBand or RoCE bypasses the kernel and writes directly to remote GPU memory, combining high bandwidth with low latency. InfiniBand offers sub-microsecond latency, while RoCE operates in the low-single-digit microsecond range. Those characteristics alone do not establish what a production deployment can guarantee under load.

RoCEv2 requires an end-to-end lossless underlay configured with PFC, ECN, and DCQCN. Colocation operators can specify dedicated ConnectX-7 InfiniBand or 400G RoCEv2 throughout the path. Managed hyperscaler fabrics, including EFA on AWS and Virgo Network on GCP, do not give the operator equivalent authority to enforce lossless behavior or inspect fabric utilization.

For reliable sub-2-µs cross-node KV transfers at sustained throughput, that distinction becomes a deployment constraint. Ultra Ethernet 1.0 offers another consideration for new colocation builds. Its physical layer uses existing Ethernet cabling and transceivers, but its new transport protocol requires UET-capable endpoints.

Where hyperscaler managed cloud is still the right answer for inference workloads

Early production, seasonal demand, and large changes by time of day can leave dedicated hardware idle enough to undermine its economics. Cloud’s zero-capex, zero-lead-time model avoids that stranded capacity. When serving fits within a single node and needs no disaggregation, there is also less reason to own a custom interconnect.

Operational capacity is a separate constraint. SageMaker, Vertex AI, and Azure ML reduce the work of self-hosting vLLM, managing NVMe tiers, and maintaining InfiniBand fabric. For a team serving a single model at moderate concurrency, that simplification can offset a higher GPU-hour price. Hyperscalers also have an edge when users are spread around the world: their presence in dozens of regions puts them close to users, while a single colocation site needs more investment to match that reach.

Hardware availability can also favor a cloud deployment. AMD MI300X, which some GPU cloud providers offer, has higher memory bandwidth than some NVIDIA hardware. For memory-bandwidth-limited decoder-only inference, that can produce a performance-per-dollar advantage for particular model architectures.

The storage layer as the variable most teams fail to measure before committing to a deployment model

Storage throughput is often absent from an otherwise detailed comparison of GPU-hours and egress. The deployment must sustain NVMe reads, KV serialization, and cache retrieval at production concurrency. A cache can fit on disk and still fail to save the intended compute if retrieving it takes longer than the serving workload’s latency requirements allow.

KIVI and KVQuant use per-channel key quantization and per-token value quantization, with representations down to 2-bit, reducing the I/O bandwidth required for long-context KV data. The storage path must also handle AI-native formats such as BF16 and FP8 without general-purpose compression paths adding latency. CacheGen takes a different approach through a custom tensor encoder that produces compact bitstreams with minimal decoding overhead.

The backend must keep up with those compressed reads and writes. Its required throughput should therefore be evaluated at the compressed bitstream rate. This is an interaction between the serving stack and the storage system. A framework benchmark that omits this path leaves a central part of production inference unmeasured.

Instrument KV cache hit rate, cold block retrieval latency, and NVMe read throughput under peak concurrency on a representative workload. These measurements reveal whether reuse actually relieves compute pressure or simply moves the bottleneck. Both deployment models fall short of their claimed cost advantage when storage keeps the GPUs from running inference.

A practical evaluation framework for matching deployment model to workload profile

The evaluation starts with four workload and deployment variables. They are sustained GPU utilization, KV working set size relative to HBM, cross-node KV transfer bandwidth, and whether the NVMe tier is configurable. The first establishes the economic regime, while the others test whether the environment can support the serving architecture. Storage measurements then show whether the planned hierarchy can keep up at production concurrency.

Colocation is the stronger choice when sustained utilization exceeds 55–60%, disaggregated serving requires guaranteed lossless transfers, and long contexts make NVMe offload routine. The engineering team must also be able to operate the storage and network stack. Cloud fits variable utilization, geographically distributed endpoints, and serving architectures that remain within one node. When those stages have different needs, a hybrid split makes sense, with cloud for experimentation and bursty training and dedicated hardware for sustained inference.

Serving software belongs in the same evaluation. HuggingFace’s Text Generation Inference moved into maintenance mode in December 2025, and its repository was archived as read-only on March 21, 2026. Teams still using TGI should factor migration to vLLM or SGLang into infrastructure planning because framework choice affects access to PagedAttention, RadixAttention, and LMCache integration.

MLCommons published MLPerf Inference v6.0 in April 2026, adding a GPT-OSS 120B open-weight benchmark for mathematics, scientific reasoning, and coding. Its results give teams a baseline for comparing vendors before they test their own workloads. They do not, by themselves, prove that an open-source stack is faster or cheaper than a proprietary model provider for a particular service. That conclusion requires the actual workload comparison, with utilization, KV pressure, transfer requirements, and storage throughput measured before the deployment model is chosen.