Managed LLM Inference APIs vs. Self-Hosted Open-Source Models
The self-hosted crossover depends on utilization, not just GPU costs.
Choosing between managed LLM inference APIs and self-hosted open-source models is an infrastructure engineering decision. Open-source LLM inference stack benchmarking and optimization must compare sustained token volume, achieved GPU utilization, latency, KV cache behavior, and operational cost against API billing. Self-hosting becomes cheaper only when useful throughput offsets its fixed costs, while data residency can make it necessary before that crossover.
Why this decision is harder than it looks
“GPU rental is cheap, API tokens are expensive, therefore self-host” is an understandable engineering instinct. It leaves out the decisive inputs, however. GPU rental is only part of the fixed cost, and available capacity is not the same as achieved utilization. A deployment that looks cheap at full throughput can cost too much when its GPUs sit idle.
The choice also includes three deployment modes. Closed commercial APIs from OpenAI, Anthropic, and Google bill for each token and spare teams from owning infrastructure, but cost the most to switch away from. Managed open-weight inference from Together AI, Fireworks, Groq, and Baseten is as easy to use as an API, with model weights that teams can move and costs per token below those of frontier closed APIs. Fully self-hosted deployments run open-weight models on rented or owned GPUs, so teams have full control over the data and take on all the work of running them.
How the two cost structures actually differ
Hosted API cost scales close to linearly with token volume. With self-hosting, teams first pay for GPU capacity and the work needed to run it. Those costs start before the deployment serves any traffic. Within that capacity, the marginal cost of another token is near zero.
This asymmetry makes utilization the denominator of the self-hosted calculation. The cost per token falls as a deployment serves more tokens for the same fixed budget. That decline stops at the serving framework’s throughput limit. Beyond the available capacity, the deployment needs a different infrastructure budget. A valid comparison therefore needs both total operating cost and achieved throughput.
Where the break-even actually sits, and why it moves
The crossover occurs when API spending exceeds the cost of self-hosted capacity that can actually serve the workload. A more expensive API rate brings the crossing closer, while a cheaper API rate pushes it farther away. Greater operational overhead moves it outward again, and higher useful GPU utilization moves it inward.
This makes comparisons against frontier closed APIs different from comparisons against budget-tier APIs. A deployment may justify its fixed costs against the former without becoming economical against the latter. Traffic shape can change the answer as well. High projected peaks do not establish the sustained throughput needed to amortize GPU capacity, and long contexts can consume memory that would otherwise support more concurrent requests. Break-even depends on workload, serving efficiency, and token volume.
KV cache is the infrastructure variable most cost models ignore
KV cache stores key and value vectors from previous tokens so that inference avoids redundant computation. Its footprint grows linearly with context length. At long contexts, the cache can exceed a single GPU’s entire HBM capacity before concurrency becomes substantial. A large Llama 3.1 model at 128K context requires roughly 40 GB of KV cache per concurrent user. Eight concurrent users require roughly 320 GB, against an H100’s 80 GB of HBM.
Managed APIs absorb cache management, batching strategy, and speculative decoding into per-token pricing. Self-hosted teams must implement these optimizations or accept substantially worse throughput. The available approaches operate at different layers. vLLM’s PagedAttention uses OS-inspired paging to eliminate fragmentation, achieving near-zero wasted KV cache space. Prefix-sharing systems use radix trees and dynamic eviction, while CachedAttention reuses cache across dialogue turns.
Model architecture also changes the memory requirement. Multi-Query Attention shares key/value heads across all attention heads, Grouped-Query Attention shares them within groups, and DeepSeek’s Multi-Head Latent Attention uses low-rank projections. Format choice is another direct lever on concurrency. FP8 KV cache with FP8 attention is nearly lossless on reasoning benchmarks, while NVFP4 uses a substantially smaller footprint than BF16 or FP8. On a B200 SXM6, moving from BF16 to FP8 and then adding NVMe offload can stack concurrency gains multiplicatively.
GPU memory hierarchy determines what KV cache optimization is actually possible
Cache optimization depends on where blocks reside and how quickly they can move. The relevant hierarchy spans GPU HBM3, CPU DRAM, CXL 3.0 memory pools, NVMe through GPUDirect Storage, RDMA fabric, and parallel filesystems. Bandwidth and latency differ by orders of magnitude. Teams must decide which tokens deserve scarce GPU memory and which can sit farther away without undermining serving performance.
A production tiering pattern keeps hot blocks in VRAM, warm blocks in CPU DRAM, cold blocks on local NVMe, and shared or persistent blocks in networked storage. Layers such as LMCache manage that placement. Cache-aware serving has reported time-to-first-token gains of 3–10× on cache hits in repeated-context workloads. Multi-turn chat, RAG with repeated system prompts, and agentic loops can benefit because stored context avoids repeated prefill computation. At current context lengths, retrieving KV cache from NVMe is routinely faster and cheaper than recomputing prefill.
NVIDIA’s Inference Context Memory Storage Platform, announced at CES 2026, standardizes KV cache offload to NVMe SSDs. It uses BlueField-4 DPUs to move cache I/O off the GPU compute path, keeping GPU SMs focused on computation. This makes offload a first-class infrastructure design choice. API consumers do not choose the storage tier or movement mechanism, but self-hosted teams must account for both when sizing concurrent capacity.
Serving framework choice sets the throughput limit before any other optimization
vLLM, TensorRT-LLM, and SGLang produce meaningfully different throughput and latency profiles on identical hardware. There is no workload-independent ranking among them. SGLang’s roughly 29% throughput advantage over vLLM when requests share context illustrates how application behavior can favor a particular engine. That result applies to multi-turn applications, though the ranking can change with the workload. The choice of framework can thus change the cost per token while the GPU budget stays the same.
Disaggregated prefill/decode separates compute-intensive prefill from memory-bandwidth-intensive decode across different GPU pools. NVIDIA Dynamo enabled a substantial throughput increase over traditional serving on a large Llama 3.1 interactive benchmark in MLPerf. The gain depends on infrastructure beyond the engine itself. NVLink supports single-node GPU-to-GPU KV transfer, InfiniBand RDMA supports multi-node disaggregated serving, and PCIe supports local GPU-to-CPU-to-NVMe movement. The NVIDIA Inference Xfer Library handles point-to-point RDMA using UCX, making the transfer path part of the serving design.
Lossless compression of KV cache tensors is an underused efficiency lever
Quantization reduces cache footprint by changing numerical representation, while lossless compression preserves the represented values. KV cache tensors offer structure that makes this possible. BF16 uses 1 sign bit, 8 exponent bits, and 7 mantissa bits. Sign and mantissa bits have near-uniform distributions in practice, but exponent bits are highly non-uniform. Entropy coding can target that exponent skew rather than treating every bit as equally compressible.
Intel research extending the ZipNN approach separated exponent and mantissa components and compressed them independently, extending the method to FP8 and FP4 formats. The research found meaningful compression for BF16 and FP8, including inference-time memory reductions for BF16 KV tensors and higher compression ratios for FP8 caches. DFloat11, presented at NeurIPS 2025, claimed lossless LLM compression through dynamic-length float encoding with full accuracy preservation. NVIDIA’s nvcomp library provides a production reference for GPU-side lossless compression. These examples concern distinct implementations.
For self-hosted inference, the consequence is more effective memory capacity without quantization loss. More hot cache blocks can remain in HBM before eviction to NVMe. That reduces I/O pressure and improves time to first token on cache hits. Compression therefore connects directly to the tiering decision, rather than serving only as a technique for shrinking model checkpoints.
Data residency and compliance requirements often decide before the cost math does
In healthcare, finance, and legal workloads, compliance rules can rule out deployments that would otherwise save money. HIPAA mandates, GDPR data residency rules, and attorney-client privilege concerns can make self-hosting or a private cloud endpoint necessary regardless of token cost. The economic comparison then has a narrower scope. It concerns the available compliant deployments.
Managed cloud AI offers a middle path for enterprises with existing cloud estates. Azure OpenAI, AWS Bedrock, and GCP Vertex AI provide frontier models within a cloud boundary, with VNet integration, private endpoints, IAM controls, and enterprise SLAs. That setup costs more than direct API access because it adds a pricing markup and infrastructure overhead. Managed open-weight inference is different because data still travels to a third party’s infrastructure. To keep data within the organization’s boundary, teams need a fully self-hosted or private-cloud deployment.
What MLPerf results reveal about infrastructure performance ceilings
MLPerf Inference benchmarks provide a credible public reference for expert-tuned infrastructure at scale. They do not predict an ordinary deployment’s throughput. MLPerf Inference v6.0 included four GB300 NVL72 systems interconnected with NVIDIA Quantum-X800 InfiniBand, achieving a record tokens-per-second result on DeepSeek-R1 offline. It was the largest-scale submission to the benchmark at that time. The result establishes what GPU ownership combined with coordinated hardware and serving engineering can achieve.
MLPerf Inference v6.1, released in September 2026, recorded nearly 5.8 million tokens per second in the GPT-OSS offline test. NVIDIA’s Vera Rubin NVL72 platform debuted with up to 2.5× better throughput on DeepSeek-R1 and up to 3.7× better throughput on Qwen3-VL than the prior generation. These results concern specific models and benchmark conditions. Offline throughput is not an interactive latency guarantee, so the figures cannot establish that an arbitrary self-hosted application will outperform a proprietary API.
Cisco submitted the benchmark’s first cross-vendor heterogeneous accelerator deployment, combining NVIDIA H200 and AMD Instinct MI350X GPUs through a Cisco G200 Network. Mango LLMBoost delivered the first prefill/decode-disaggregated results on AMD Instinct GPUs and the largest multi-region cluster submitted, spanning four sites on two continents. That cluster operated as one endpoint at very high scaling efficiency. Such results reveal headroom from orchestration and interconnect design, reinforcing why framework benchmarks alone cannot describe an entire inference stack.
How should teams benchmark their workload before choosing an API or self-hosting?
Teams should first work out their data residency constraints, then compare costs. Once teams know which deployment modes they can use, they should set up tools to measure sustained token volume and actual API spending, keeping projected peaks out of that measurement. Achieved GPU utilization on similar workloads supplies the corresponding self-hosted input. Low utilization inflates cost per token more than any other single factor. Operational capacity belongs in the same calculation because a specialized GPU infrastructure engineer is a substantial fixed cost.
The workload profile should show how context length varies and how requests share context. Long contexts determine cache memory requirements and whether NVMe offload or architectural choices such as MQA, GQA, or MLA become necessary. Shared prefixes change the value of prefix caching. For multi-turn chat, RAG, and agentic loops, that can make self-hosting competitive at lower volumes and make the SGLang-versus-vLLM comparison especially relevant. Throughput, latency, and cache-hit behavior must describe that workload.
For benchmarking tools and engines, MLPerf supplies the public infrastructure reference, while vLLM, TensorRT-LLM, and SGLang are the self-hosted candidates whose workload-specific behavior determines the comparison. No cited engine result alone proves that open-source inference is faster and cheaper than OpenAI or Anthropic at scale. That conclusion requires measured useful throughput, latency, and full operating cost against actual API spending.
The recommended progression is to start on a commercial API with token-spend instrumentation, move to managed open-weight inference as volume grows, and adopt full self-hosting when the numbers or compliance requirements justify it. The intermediate tier should not be skipped. When self-hosting becomes warranted, KV cache format, NVMe offload, serving framework, prefix caching, and lossless compression become first-class engineering decisions because they determine how much useful service the GPU budget buys.
