NVMe Storage Costs vs. GPU Memory for Inference Offloading

LLM inference now eats roughly 80% of AI infrastructure budgets, and by some measures accounts for 78.4% of all accelerated compute spend in 2026. That figure alone should settle an old argument: serving is the dominant cost problem in AI, not training. And the single biggest driver of serving cost, once a model is deployed, is memory, specifically the KV cache, which stores the key and value tensors from every prior token, every layer, every attention head, so the model never has to recompute context from scratch at each decode step.

Inference splits into two phases with very different appetites. Prefill processes the entire input prompt in parallel, so it's compute-bound, GPUs chew through it fast. Decode generates one token at a time, autoregressively, and every single step needs to read the full KV cache back from memory. That makes decode bandwidth-bound, not compute-bound. The GPU sits there waiting on memory, not math.

At scale, that waiting gets expensive fast. Spheron estimates that a single user running a 128K-token context on a standard multi-head attention model needs about 40 GB of HBM just to hold the KV cache in BF16. Eight concurrent users at that same context length need roughly 320 GB, which is four times the total HBM capacity of a single H100. Multi-Head Latent Attention compresses a 503 GB per-stream requirement down to 17.3 GB in FP16, or 8.6 GB in FP8, a reduction of more than the vast majority of the original footprint. But MLA is architecture-specific, and most open-source models running in production today don't have it. Pushing the context window out to 1M tokens brings a single user's KV cache to roughly 320 GB at BF16, more than any GPU shipping today can hold on its own.

None of this is a corner case. At production-scale context lengths, running out of HBM isn't an edge condition to plan around, it's the default outcome. Where the overflow goes and what that costs is the real engineering question. It's where the overflow goes, and what that costs.

What HBM costs: hardware pricing, cloud rental rates, and the memory that GPU compute is really selling

HBM doesn't trade on any public market. Foundries and GPU makers negotiate it under confidential contract, so every number that circulates publicly, TrendForce, SemiAnalysis, Stanford's memory pricing research, is a model, not a transaction record. No one should treat an HBM price quote as gospel.

Even the modeled numbers are stark. An HBM3E stack runs several times more than the cost of a comparable amount of ordinary DDR5. Part of the gap is physical: a gigabyte of HBM takes up roughly the wafer area of four gigabytes of standard DRAM. That's not a temporary shortage working itself out; it's a structural constraint baked into how the chips get made. HBM for AI accelerators already eats up 23% of the world's DRAM wafer capacity, and AI data centers are on pace to consume 70% of all memory chips produced globally in 2026.

That scarcity raises what a rack costs. An 8-GPU H100 SXM5 80GB server runs into the mid-hundreds of thousands of dollars as of Q2 2026. NVIDIA's factory-integrated DGX H100, which bundles 30 TB of NVMe and 2 TB of DDR5 alongside eight ConnectX-7 InfiniBand adapters, lists in the high hundreds of thousands of dollars. Stepping up to H200, with its larger 141 GB HBM footprint per GPU, brings an 8-GPU box to a noticeably higher price range than the H100 server.

Renting instead of buying doesn't dodge the cost, it just spreads it out. H100 SXM rents for $3.49 an hour on Runpod, and across GPU cloud providers generally the range runs several times wider on the high end, climbing higher still on the major hyperscalers. AWS's p5.48xlarge, packing eight H100 80GB cards and 30 TB of local NVMe, runs into the tens of dollars per hour on demand. Newer silicon carries its own rate card: B200 goes for $6.79 an hour on Runpod, and AMD's MI300X, with its unusually large 192 GB of VRAM, rents for $2.39 an hour.

Now put those numbers next to the KV cache math. At the range of hourly rates typical for an H100, and a single 128K-context user chewing through roughly 40 GB of an 80 GB card, half the GPU's memory is committed to one session's cache before a single FLOP of useful compute has run. The effective cost of a concurrent user slot, once memory pressure is accounted for, is far higher than the headline hourly rate suggests. This is why newer hardware with bigger HBM pools shows such lopsided efficiency gains: B200 delivers 70B-model output tokens at a small fraction of the per-million-token cost of H100 SXM5, a multiple-fold drop driven mostly by the fact that 192 GB of HBM3e holds more cache without needing to offload anything.

NVMe pricing in 2026: the supply crisis that changes the cost math

The same wafer math that makes HBM expensive is now making NVMe expensive too. HBM production for AI accelerators consumes 23% of global DRAM wafer capacity at roughly a 3:1 wafer ratio against DDR5. As fabs shift capacity toward HBM and enterprise DRAM to chase that demand, NAND supply tightens right along with it, and SSD prices follow.

Industry analysts tracking Q2 2026 have flagged sharp quarter-over-quarter increases in both conventional DRAM and NAND Flash contract prices. Reports from late 2025 indicated that leading HBM suppliers had their 2026 production largely spoken for well in advance. Controller vendors and storage integrators have similarly reported constrained NAND availability stretching through 2026. Gartner has a name for the phenomenon now, "memflation," and forecasts a combined surge of around 130% across DRAM and SSD pricing by the end of 2026.

The consumer-facing numbers make the trend concrete. A 1TB consumer Gen 4 NVMe drive cost noticeably more as of March 2026 than in mid-2025, working out to roughly nine to ten cents per gigabyte. A 2TB drive costs somewhat more than the 1TB drive, scaling up accordingly. Enterprise-grade 30 TB TLC SSDs tell an even sharper story: $3,062 in mid-2025 climbed to $17,500 by early 2026, a 472% increase in under a year.

Relief isn't close. Industry outlooks suggest the DRAM supply gap could persist for several years, with NAND easing somewhat sooner, though a full return to pre-shortage pricing is not expected to be near-term.

Even with all that, NVMe hasn't lost its fundamental cost advantage over HBM, per gigabyte it's still cheaper by orders of magnitude for caching-tier workloads. The gap has just narrowed. And that narrowing matters for anyone doing capacity planning, because the old assumption, that NVMe is basically free next to GPU cost, doesn't hold in 2026 the way it did a couple of years ago.

The four-tier memory hierarchy and what each tier delivers for KV cache traffic

NVIDIA's reference architecture (covered under its ICMSP framing, announced at CES 2026) lays out four tiers, each with a distinct latency and bandwidth profile, and each suited to a different age of KV cache data.

G1 is GPU HBM itself: nanosecond access, holding the active KV data for whatever token is being generated right now. H100 SXM delivers 3.35 TB/s of bandwidth here; B200 more than doubles that, at 8.0 TB/s. G2 is CPU DRAM, reached over PCIe, roughly 63 GB/s on a PCIe 5.0 x16 link per Spheron's figures, at microsecond latency, holding warm KV data from sessions that just finished but might resume. G3 is local NVMe SSD, sub-millisecond to low-millisecond latency, delivering around 7 GB/s of effective throughput on PCIe Gen 4, and it holds cold KV: historical context, prefix caches, anything unlikely to be needed in the next second but too valuable to discard. G4 is networked or object storage, the slowest tier, used for prefix caches that need to persist or be shared across multiple nodes.

Production deployments tend to tier by how recently a session touched the GPU. Production field guidance for this kind of system typically keeps the most recently active KV data in HBM, pushes idle sessions to DRAM, and spills further-idle sessions down to NVMe.

Restore time matters just as much as raw bandwidth when a session comes back to life. Pulling a 128K-token KV cache for a 70B model back from DRAM over PCIe takes a meaningful fraction of a second given the bandwidth constraints of the interconnect. The same restore from NVMe, limited by the lower effective throughput of that tier, takes noticeably longer still. That gap is the whole ballgame for interactive latency: a user who paused mid-conversation and comes back a minute later is going to feel that 2-to-3-second lag if their session got pushed to the NVMe tier.

Researchers at Vrije Universiteit Amsterdam and IBM Research, publishing through ACM at CHEOPS '25, ran a detailed I/O characterization of KV cache offloading on DeepSpeed and FlexGen and found some things that cut against intuition. Offload I/O is dominated by small, 128 KiB block requests on both reads and writes. Read bandwidth badly outpaces write bandwidth, 2.0 GiB/s for reads against just 11.0 MiB/s for writes on average. The write path, not the read path, is the actual bottleneck when spilling KV data to storage. The same study found that model weight offloading barely touches the limits of an NVMe SSD, it's KV cache offloading specifically that stresses the storage layer. Libaio-based tensor offloading beat plain POSIX I/O on both reads and writes, making I/O library selection a first-order performance decision, not an afterthought.

Hardware vendors are racing to fix the transport layer directly. NVIDIA's Dynamo framework moves KV blocks using its NIXL library, choosing NVLink for GPU-to-GPU transfer within a node (900 GB/s bidirectional on H100 SXM with NVLink 4.0), InfiniBand RDMA across nodes, PCIe for the local tier, and TCP as a fallback, and it plugs into both vLLM and SGLang. NVIDIA's ICMSP architecture, announced at CES 2026, goes further, using BlueField-4 DPUs to pull KV cache movement off the GPU's own compute path. Deployments pairing ICMSP with an all-NVMe storage backend have reported prefill times several times faster. This acceleration needs BlueField-4 DPUs physically installed inside a dedicated ICMS storage enclosure, a separate rack from the GPU compute servers, not a software toggle on existing hardware.

Software paths for NVMe offloading and the I/O overhead each approach carries

Hardware acceleration is only available to teams willing to buy the dedicated rack. Everyone else is working the problem in software, and the quality of that software varies enormously.

LMCache adds persistent storage backends to vLLM's existing in-process prefix cache, and works across vLLM, SGLang, and NVIDIA Dynamo, sharing the same NIXL primitives Dynamo uses for moving KV blocks around. Persistent storage backends replace vLLM's built-in --swap-space flag, which is limited in scope compared to a persistent, multi-replica caching layer. LMCache, by contrast, supports NVMe, Redis, and remote storage backends, survives restarts, and can share cache across replicas with a configurable chunk size. As BlueField-4 infrastructure becomes more common, LMCache picks up NIXL-accelerated paths, but it already implements the same tiered pattern in pure software over PCIe on hardware that's sitting in racks today.

A framework called DUAL-BLADE, described in an April 2026 paper (arXiv:2604.26557) from researchers at Sogang University, Samsung Electronics, and Florida State University, targets a different problem: edge AI systems running on 8 to 32 GB of DRAM with a single GPU and a genuinely tight memory budget. The paper identifies three specific ways naive file-based NVMe offloading fails under decode's read pattern. Page-cache thrashing, because decode's autoregressive access pattern doesn't play nicely with a general-purpose page cache. Excessive overhead moving data through the VFS, filesystem, and block layer stack. And loss of sequential locality at the device level, because the kernel's queue architecture can scatter what should be sequential reads. DUAL-BLADE's fix splits KV tensors into two paths: whatever fits inside the DRAM page-cache budget goes through the normal page-cache path, and anything beyond that budget goes through an NVMe-direct path using contiguous logical block address regions, bypassing the filesystem layer. The reported results are substantial: prefill latency down by a substantial share, decode latency down by a substantial share, and SSD utilization improved several times over across a range of memory budgets.

DeepSpeed and FlexGen, the two frameworks the CHEOPS '25 team studied directly, both support offloading model weights and KV cache to SSD, and that same study found a gap between two different I/O access methods. Async I/O through libaio consistently beat standard POSIX calls, a tuning knob a lot of deployments apparently leave untouched.

GPU Direct Storage, built around NVIDIA's cuFile library and the broader ICMSP architecture, removes the CPU staging step from the path between NVMe and GPU memory. That matters most on the write side of KV cache spillover, and in any high-concurrency setup where PCIe bandwidth is the shared resource everyone's competing for.

Placed side by side, the four approaches show a wide spread. "NVMe offloading" as a label covers everything from a naive mmap-based design that thrashes the page cache under real load, to a kernel-bypass, direct-NVMe path running close to the physical bandwidth ceiling of the device. The hardware tier a team picks matters, but the software sitting on top of it matters just as much, sometimes more.

Building the TCO framework: cost-per-token as the unit that unifies HBM and NVMe decisions

Figures cited by spendark.com show per-token inference cost collapsing by several orders of magnitude over three years, from around $20 per million tokens down to a small fraction of that. And yet total inference spending keeps rising, because volume is growing faster than unit cost is falling. That combination points to the right optimization target: not raw hardware price, not even cost per GPU-hour, but cost per token at a defined quality-of-service level. Everything in the memory hierarchy, HBM versus DRAM versus NVMe, is really a lever on that one number.

Three variables interact to set that number. Memory capacity determines how many concurrent users a deployment can serve before something has to be evicted. Bandwidth determines how fast a cold KV cache can be pulled back into HBM once a session resumes. And latency tolerance determines how much restore delay a given use case can actually absorb before users notice.

For short-context, low-concurrency workloads, contexts under roughly 32K tokens with only a handful of simultaneous users, NVMe offloading doesn't buy anything. There's no capacity pressure to relieve, so the added I/O latency is pure overhead with no offsetting benefit. Full HBM serving is simply cheaper per token in that regime, and the offload path sits there unused.

Long-context, high-concurrency serving, 128K tokens and up with many sessions running at once, reverses the picture completely. HBM-only serving isn't just more expensive there, it's physically impossible on a single H100: 80 GB of HBM total, roughly 40 GB consumed by one 128K-context BF16 user, means a second concurrent user alone exhausts the card. At that scale, NVMe offloading isn't an optimization a team chooses to adopt, it's the only route to serving multiple users without resorting to tensor parallelism spread across additional GPUs, which brings its own cost and complexity.

Between those two extremes sits a crossover point, somewhere context length and concurrency combine to make offloading worth its I/O tax. Where exactly that point falls depends on the model's attention architecture, the restore latency a given application can tolerate, and the price gap between HBM and NVMe at the moment a team is deploying, a gap that, as the 2026 supply numbers make clear, is not fixed. Finding that crossover for a specific workload is the actual engineering task. Everything else, the hardware pricing, the tiering diagrams, the I/O library benchmarks, exists to help a team locate it.