The Infrastructure Variables a Rigorous Open-Source Inference Benchmark Cannot Omit

A rigorous open-source LLM inference benchmark must control the inference engine, storage I/O, KV cache tiering, network fabric, and tensor transfer paths. Leaving these variables unspecified makes throughput and latency results non-reproducible across deployments. For open-source LLM inference stack benchmarking and optimization, the infrastructure specification is part of the experiment.

Why inference benchmarks that ignore infrastructure cannot transfer to production

An impressive throughput number may not hold up when the same inference engine runs on a practitioner’s own cluster. The storage stack, memory tiers, and transfer paths have changed even if the engine remains the same. GPU utilization cannot explain that difference alone. py-kvcache research found that transfer granularity, intermediate memory use, and the point at which transfers enter the request schedule affect cache performance independently of device bandwidth. Two runs with the same GPU utilization can thus have different time to first token, or TTFT.

An engineering team cannot establish whether published throughput applies to its deployment when the benchmark omits the conditions that produced it. This also limits what an engine comparison proves. Results credited wholly to inference software may come in part from differences in cache retention, storage access, or network transport.

Why KV cache memory pressure is the forcing function behind all storage variables

KV cache size grows linearly with context length, batch size, model width, and number of layers. GPU high-bandwidth memory, or HBM, must accommodate that growth. HeadInfer research shows that at 1 million tokens, a Llama-3 model’s KV cache alone requires more than the entire HBM of an H100. Long-context serving consequently becomes a storage problem as well as a compute problem. The cache cannot remain entirely in GPU memory.

Prefill is compute-bound and determines TTFT, while decode is memory-bandwidth-bound. Once the working set is larger than HBM, external cache access can become part of the critical path for serving requests. Reuse is not automatically faster than recomputation. A benchmark must establish whether restoring a KV block costs less than recomputing its prefix, given block size, concurrency, storage tail latency, context length, and retained cache capacity.

The interconnect also constrains that break-even point. Understanding Bottlenecks for Efficiently Serving LLM Inference With KV Offloading identifies severe PCIe bandwidth bottlenecks when KV caches are offloaded to CPU DRAM. More capacity does not ensure that the cache can be restored faster. The benchmark must therefore characterize both where the cache resides and how it moves back to the GPU.

The KV cache tier configuration as a controlled benchmark variable

Red Hat OpenShift AI 3.5 benchmarks demonstrate how tier selection changes the result. The tests used Nemotron-Ultra on H100s with agentic workloads averaging roughly 50K input tokens. CPU offload alone delivered a throughput gain, while adding NVMe or CephFS as a secondary tier nearly doubled that gain. Without offload, mean TTFT was many times longer. At high concurrency, the external cache hit rate for the CPU-only tier fell sharply, while hit rates with CephFS offload stayed much higher.

The payoff from that additional capacity depends on the workload. py-kvcache found that asynchronous direct I/O and scheduler-aware preloading improved disk loading for long contexts. On an H100 with shorter average requests, however, GPU memory retained enough prefixes for disk offloading to fall below break-even. The same tier could help or hurt. Request length distribution must therefore accompany the tier configuration.

Different tier topologies also represent distinct experiments. A single-node path through GPU HBM, CPU DRAM, and local NVMe preserves locality but limits capacity to the node. A shared POSIX filesystem introduces higher per-block latency while enabling cross-replica reuse and persistent capacity. An S3-compatible object tier represents another path, relevant to disaggregated prefill/decode architectures. Each topology requires separate reporting, including connector configuration, admission rules, and eviction policy, because counting every prefix hit as beneficial can overstate reuse gains. Even then, topology explains only some of the differences, because the access mechanism within each tier is a separate variable.

The I/O path within each tier — io_uring, GPUDirect Storage, and NVMe-direct as distinct benchmark parameters

“NVMe” does not specify how an application accesses data. Standard filesystem access, io_uring, NVMe-direct, and GPUDirect Storage remove different software overheads. DUAL-BLADE research found that file-backed mmap/page-cache access caused page-cache thrashing under decode’s autoregressive read pattern, producing unpredictable latency. Its direct-to-LBA path bypassed the filesystem, substantially reduced prefill and decode latency, and improved SSD utilization. Both paths used the same NVMe hardware.

Scheduling introduces another distinction within the access path. py-kvcache traced KV blocks through vLLM’s scheduler, connector interfaces, CPU and GPU memory, and filesystem storage. Scheduler-aware preloading started disk reads while requests were still queued and contributed a measurable improvement independently of device bandwidth. A specification must name the access mechanism, driver layer, and queue depth, and state whether transfers are synchronous or asynchronous relative to the scheduler.

GPUDirect Storage, or GDS, removes CPU staging from the NVMe-to-GPU path by allowing direct DMA into GPU memory. Local NVMe uses the device’s DMA engine without a network, while remote or distributed GDS requires an RDMA-capable NIC. These are different hardware prerequisites. For small random inference I/O, io_uring offers low latency and competitive IOPS, while throughput-intensive phases such as checkpoint writes favor GDS. A benchmark must distinguish those access patterns and declare whether its remote path depends on an RDMA fabric.

Network fabric as a hardware prerequisite that benchmark specifications must declare

RDMA-dependent paths cannot be reproduced on a TCP/Ethernet cluster without RDMA support. GDS over NVMe-oF and GPUDirect RDMA-based KV transfer require suitable hardware, including InfiniBand or RoCE networking and GPUDirect RDMA-capable NICs. A result obtained through those paths does not transfer to a cluster that lacks their prerequisites, even if its GPUs and storage devices match.

Among supported configurations, network performance still changes the outcome. Disaggregated inference moves KV blocks between prefill and decode workers, so effective fabric bandwidth affects prefill latency. High-bandwidth InfiniBand and 25 GbE are not equivalent test conditions. At high cluster scale, uncontrolled RDMA traffic can also create incast congestion and degrade tail latency despite sufficient nominal bandwidth. Because the 99th percentile is a primary inference SLO metric, a specification must declare congestion control, such as DCQCN or a custom paced scheme, and its parameters.

The network declaration must also include medium, link speed, GPUDirect RDMA status, and any NVMe-oF transport. Those fields show which paths are available and the conditions in which KV moves under load. They do not, however, establish how many bytes must move. Tensor representation remains a separate bandwidth variable.

Tensor encoding and lossless compression as a variable that shifts apparent bandwidth requirements

Uncompressed BF16 KV tensors demand more transfer bandwidth than the same data encoded with effective lossless compression. A benchmark using only the uncompressed representation can therefore overstate the bandwidth required by a compressed deployment. GAAC, the Group-Adaptive Additive Codec, exploits exponent redundancy in BF16 tensors and achieves consistent lossless compression ratios across multiple tested models.

Inference latency gains from those savings depend on integration. ZipServ, presented at ASPLOS 2026, combines Tensor-Core-Aware Triple Bitmap Encoding with a fused decompression-GEMM kernel, ZipGEMM. Cloudflare’s Unweight April 2026 technical report describes storage and transfer savings through Huffman coding with byte-level separation of BF16 exponents, plus reconstructive matmul kernels that decompress weights in on-chip shared memory. These are distinct integration choices. They show why a codec name alone cannot characterize compression’s effect on the inference path.

A full declaration must state both how compression works and what it costs to run. It must name the codec, state whether encoding is lossless or lossy, identify CPU or GPU integration, and explain whether decompression blocks execution or overlaps asynchronously. Lossless compression preserves the KV tensor values. Lossy compression alters those values and the model’s output distribution, so an undeclared lossy configuration is effectively a different model experiment. Public storage benchmarking has already begun formalizing some of the surrounding infrastructure variables, although not this full set.

What MLPerf Storage v3.0 formalizes and what open-source inference benchmarks still omit

MLCommons announced MLPerf Storage v3.0 results on September 1, 2026. The version added an S3 data layer to the existing POSIX layer, together with KV cache and vector database workload tests. MLPerf Storage working group co-chair Curtis Anderson said that as AI context scales reach into the trillions, object-based storage may become a workable, and perhaps preferred, choice in place of filesystem-based storage.

These additions establish concrete reference points for inference benchmarking. A test restricted to local NVMe or POSIX filesystems covers only part of the deployment space represented by the standard. End-to-end throughput and TTFT also cannot substitute for isolating the KV cache I/O pattern when comparing against storage results. Architecture declarations are part of that discipline. At least one v3.0 submitter declared a 5+2 erasure coding scheme for training and checkpointing benchmarks, illustrating the specificity required without establishing a universal KV-cache configuration.

Open-source inference benchmarks must additionally expose tier admission policy under real request distributions, RDMA fabric and congestion control, and tensor compression integration. Public, comparable storage benchmarks provide evidence for component evaluation, but they cannot replace measurements of how that component interacts with the inference scheduler and workload.

A complete benchmark specification checklist for reproducible open-source inference benchmarking

The practical requirement is a declaration covering every dimension below. For KV cache tiers, it must state topology and absolute capacity at each level, from HBM through CPU DRAM to local NVMe, shared POSIX storage, or S3-compatible storage. The declaration must include eviction policy, admission gate, working set relative to each tier, and cold-cache, warm-cache, or steady-state conditions.

For each tier, the declaration must identify the access mechanism and driver layer, including page-cache or mmap access through kernel VFS, io_uring, NVMe-direct through SPDK or LBA bypass, or GDS through cuFile. It must also state the queue depth and I/O concurrency. Scheduler integration needs its own explicit description. That includes synchronous or asynchronous operation and whether preloading overlaps disk reads with queued requests.

The network fields must state InfiniBand generation, RoCE, or TCP-Ethernet, together with link speed, GPUDirect RDMA status, congestion control parameters, and whether NVMe-oF is used over that fabric. Tensor fields must identify the KV data type, such as BF16 or FP8, and compression status. A codec declaration needs its name and version. Lossy configurations additionally require compression ratio and quality impact, while all compressed configurations require the integration point and whether decompression is synchronous or overlapped.

The engine declaration must identify its version and KV connector name and version, such as LMCache, vLLM KV Offload API, py-kvcache, or SGLang HiCache. It must state whether prefill and decode are disaggregated and identify their KV transfer protocol. Workload declarations are equally necessary for interpreting those choices. They must include input-length mean and tail, prefix reuse rate, prefix-length distribution, concurrent request count, fixed concurrency or arrival process, and cache warm-up duration before measurement.

Distributed tiers require the erasure coding scheme, shard placement policy, GPU-to-shard placement strategy, and POSIX, S3-compatible, or custom interface. Mixed-use clusters also require checkpoint size, the write-bandwidth floor needed without degrading inference throughput, recovery time objective, and whether checkpoint and inference I/O share storage and fabric. These fields make competing explanations for a result testable. Publishing them enables cross-deployment comparison, attribution of performance differences to specific variables, and progress toward the standardization that MLPerf Storage v3.0 has begun for the storage layer.