The Infrastructure Cost Variables That Proprietary API Benchmarks Deliberately Ignore

Open-source LLM inference stack benchmarking and optimization requires measuring production cost alongside token throughput against a proprietary API. A credible comparison includes KV cache memory, context residency, storage I/O, network transfer, idle capacity, and engineering overhead at realistic concurrency. Those variables decide whether self-hosting is faster and cheaper at scale, but the usual API scorecard leaves them out.

Why API benchmarks are built around variables the vendor controls

Per-token price is the visible number in an API comparison. It is easy to compare because much of the infrastructure has been removed from view. Accuracy scores on standard evaluations and latency at idle or single-request load complete a scorecard built around conditions the provider can optimize. Short contexts, low concurrency, and no persistent state or multi-turn accumulation keep memory pressure and data movement outside the measurement.

That is a structural asymmetry. Providers control their deployed infrastructure, but not the context growth, traffic patterns, or session behavior of a customer's production workload. A quoted token rate does not show how those variables change the actual cost. At low volume, that rate can dominate the purchasing decision. Under long-context, concurrent, agentic load, memory pressure and state residency become major cost drivers, while repeated calls and growing inputs can make an apparently cheap task expensive.

How KV cache memory pressure becomes the largest unquoted cost at scale

The KV cache stores computed attention keys and values for each token and layer so that the model does not recompute them during decoding. The memory it needs grows linearly with context length, layer count, and head count. Beyond moderate context lengths, KV memory can outpace parameter memory, then dominate it. At very long contexts, cache residency and processing can consume the great majority of available GPU VRAM and wall-clock time per token.

Short-context benchmarks hide much of that pressure because they test the model before the state it has built up becomes the main constraint. Long-document, multi-turn, and agentic applications operate in precisely the opposite conditions. DeepSeek's attention mechanism stores a low-rank projection of KV state, which compresses it much more than grouped-query attention does. That difference can produce a large cost gap at very long context without appearing in a short-request API comparison.

LMCache, from Tensormesh Inc. and the University of Chicago, makes the recomputation-versus-transfer choice concrete by extracting KV state from vLLM and SGLang and sharing it across queries and engines. Reusing that state replaces redundant computation with a storage and transfer requirement. Paged attention, prefix caching, attention-layer compression, and KV quantization address different parts of this expense. FP8 KV cache halves memory consumption with minimal accuracy regression on most production workloads, while INT8 has a larger accuracy penalty on long-context retrieval, making precision a joint cost-and-quality decision.

Why prefix caching and context residency are cost variables that only appear under real load

Prefix caching hashes a prompt prefix, stores its KV state, and reuses that state when another request shares the prefix. A memory read replaces repeated computation for the shared portion's prefill. This makes prefix caching a high-leverage application-side optimization for agent loops, multi-tenant SaaS, and long-document workflows. But the state needs somewhere to stay so it can be reused. Compression and quantization make that residency problem more tractable without eliminating either its capacity requirement or its retrieval cost.

ObjectCache, from ETH Zurich, HPE Labs, and CUHK Shenzhen in May 2026, shows that accumulated reusable KV state can exceed both GPU memory and local DRAM. Remote DRAM pools preserve access speed, but make cache capacity a provisioned part of the serving cluster. The paper looks at using S3-compatible object storage instead. Its storage protocol and transfer schedule deliver KV data in the order the GPU consumes it, overlapping transfer with per-layer compute. On a high-bandwidth RoCE cluster, the approach adds only a small latency overhead for long contexts, demonstrating that storage topology is an engineering variable.

“Internet for the KV Cache” frames reuse as a content-distribution problem spanning clouds and datacenters. ObjectCache gives that framing a concrete basis by treating KV blocks as immutable after prefill, content-addressable by prefix hash, and reusable across users, sessions, and compute nodes. Placement therefore depends on reuse and retrieval economics. Multi-turn agents intensify the requirement because, by turn 30, input tokens per call can be many times those at turn 1, with much of the context stale. A single-request benchmark does not measure how that context grows or what it costs to keep it.

How GPU memory hierarchy and precision format choices create shadow costs that don't appear in quoted prices

Weight precision determines how much GPU memory a model occupies and, consequently, how many GPUs a deployment needs. An API price conceals those choices inside the provider's service economics. Self-hosting exposes them as capital and operating costs. Evaluations of the Llama 3.1 family found FP8 weights and activations effectively lossless across model scales, making the memory expense of BF16 an optional infrastructure choice for that level of quality. That finding applies to weights and activations, while the precision trade-offs above apply to KV.

Implementation can undo the expected memory saving silently. An October 2026 bug report against a production inference engine described an FP8 linear layer that dequantized weights to BF16 on the first prefill request on a CUDA device. The engine then retained that copy for its lifetime. For a model like DeepSeek V4.1-Flash, the shadow copy added roughly 10–15 GB of GPU memory.

Shared SRAM per GPU block is limited but has far lower latency than HBM. On-the-fly decompression uses that hierarchy to let losslessly compressed weights remain outside HBM until needed. Providers have already applied such optimizations or absorbed the expense of not applying them before publishing a result. The lesson for self-hosting estimates is to account for actual precision paths, compression, and resident copies, because each can change GPU count independently of the advertised model format.

What NVMe storage performance and checkpointing throughput actually cost at training and inference scale

Storage throughput affects training and inference through different mechanisms. Training workloads, particularly data-intensive models, can gain substantially from high-speed NVMe, while checkpointing adds a storage-throughput requirement to training cost. PCIe Gen5 drives can achieve peak throughput with fewer disks than PCIe Gen4. Inference, by contrast, depends primarily on GPU capability and memory bandwidth until state moves beyond GPU memory. KV offloading creates a direct connection between storage performance and serving cost.

MLPerf Storage v3.0 added KV cache and vector database workload tests in September 2026. That addition recognizes an inference-storage gap in earlier benchmarks. One submission used a large array of Solidigm D5-P5336 QLC NVMe drives as a KV cache tier, achieving hundreds of GiB/s of read bandwidth. Such an array is not incidental storage attached to an otherwise complete inference system. Under memory pressure, its I/O performance helps set the latency and cost floor for retrieving offloaded KV state.

Compression further connects storage economics to GPU execution. Another submission applied GPU-native lossless compression before writing to storage, reducing both footprint and network traffic for KV offloading. A compute-only comparison cannot reveal that interaction, just as a drive specification alone cannot describe the complete serving path. Storage benchmarks for KV workloads provide evidence that a short, resident API request does not.

How fabric and interconnect choices set a cost ceiling that inference benchmarks never reveal

The interconnect is a capital decision with workload-specific consequences. Large H100 training clusters can spend much of their cycles waiting for network operations during large all-reduce exchanges. How well the fabric performs therefore affects how long training takes as well as what the equipment costs. At 256 GPUs, the notes' comparison places InfiniBand infrastructure $212,000 above equivalent RoCEv2 Ethernet solutions. A throughput gain has to be evaluated against that additional investment.

High-speed Ethernet is sufficient for inference, whose KV transfer patterns are less latency-critical than training all-reduce operations, so InfiniBand is rarely cost-justified for that workload. This distinction becomes consequential when provisioning the cluster. As context grows and cache transfer replaces recomputation, bandwidth, latency, and network pricing help determine where reusable state should live. The API provider has already chosen and amortized a fabric, but its inference benchmark gives self-hosting engineers no basis for reproducing or pricing that choice.

The idle GPU, observability gap, and agent overhead costs that no benchmark prompt can model

KV architecture, storage tiers, and fabric establish the infrastructure design. Operational costs then accumulate continuously against that design. An API incurs no usage charge when idle, while a rented GPU costs the same whether it processes no tokens or a large volume. For variable workloads, idle time can occupy roughly 60–70% of the month. Continuous batching improves utilization substantially, but cannot remove the expense of capacity provisioned for peaks when demand falls overnight or outside business hours.

Without visibility into which services call the model, their token counts, and their latency, optimization becomes guesswork. A runaway process can account for a large share of GPU utilization without being apparent in a model-level benchmark. Production optimization also requires continuing engineering allocation as workload complexity and context length grow. Those costs belong in the ongoing operating estimate.

Agent behavior adds another recurring cost on either side of the hosting decision. Tool definitions and function schemas use input tokens on every API call, and a complex agent can spend thousands of tokens per request on definitions alone. Calling the model more than once then adds to the cost of each task. As context builds up across turns, session management becomes part of the same cost calculation as model price. Neither an idle-latency measurement nor a single benchmark prompt represents the resulting work, so per-task cost can diverge sharply from the impression created by a cheap token rate.

How do engineers compare total inference cost — and where does storage architecture sit?

A complete comparison starts with the workload's target context length and the KV memory needed to serve it. That requirement needs to be measured or estimated before comparing a self-hosted token cost with an API quote. Cache architecture comes first because compression, quantization precision, and prefix reuse control the dominant long-context resource requirement. Prefix cache hit rate then needs to be measured under realistic traffic, with requests tested together. That separates reusable state from capacity that must be provisioned without a corresponding reuse benefit.

Storage tier design follows because it determines the retrieval cost of state that no longer fits in GPU VRAM. ObjectCache and the KV distribution paper point toward placement based on bandwidth, latency, and reuse economics. Storage I/O for KV offload therefore needs to be measured or estimated in its own right. MLPerf's KV workload tests belong beside compute benchmarks in that evaluation, while LMCache provides an open-source cache layer for reuse and transfer across supported engines.

Fabric and precision choices then follow the workload. For long-context inference with high prefix reuse, spending on storage bandwidth makes more sense than spending more on fabric, while training with large all-reduce operations can be worth spending more on the network. Agentic workloads require session management and observability that hardware cannot replace. Before comparing self-hosting with API cost, the estimates still need to cover the idle GPU fraction under production traffic and the ongoing engineering overhead. Proof that an open-source stack is faster and cheaper comes from that complete workload accounting, with storage and KV architecture treated as first-class decisions and their costs accounted for separately from the token rate.