Hidden Costs of Running Open-Source LLMs in Production

The pitch sounds simple: download the weights for free, skip the per-token API bill forever, and pocket the difference. That intuition drives most self-hosting decisions, and it's wrong in a specific, measurable way. Aisuperior.com puts model weights at something like 2 to 5% of what it costs to run an open-source LLM in production. The other 95 to 98% lives in infrastructure, staffing, and the ongoing work of keeping the thing running, and that's the part almost nobody prices out before they commit.

The cost structure flips, too. A proprietary API charges by the token: variable, visible, easy to forecast against usage. Self-hosting inverts that. Most of the spend becomes fixed, and it runs whether or not a single request comes in overnight. Then there's the scale trap: a pilot pushing a moderate volume of tokens a day can turn into a workload dozens of times larger in production with zero changes to the architecture. That's a jump of several dozen times over, and fixed infrastructure doesn't stretch linearly to meet it. Putting the pieces together, aisuperior.com puts total cost of ownership at $125,000 to $820,000 a year for most organizations, at minimum. Startups report infrastructure eating 40 to 60% of revenue. That's the norm once someone actually adds it up, not an edge case. That's the norm once someone actually adds it up.

Before the bill lands, it helps to know where each piece of it comes from. Start with hardware: it's the most capital-intensive layer, and the least room exists to negotiate around it.

How GPU memory physics set a hard floor on infrastructure spend

Web apps run on cheap CPU boxes because they mostly move small amounts of data around and wait on a database. LLMs don't work that way. Before a model generates a single token, it needs its parameters sitting in High-Bandwidth Memory, and that requirement alone sets a floor no amount of clever engineering gets under.

The floor moves with model size. A 7-billion-parameter model fits on a single NVIDIA L4 with 24 GB of memory, and that card runs roughly $2,000 to $3,000 on-premise. At parameter counts well beyond that range, the math changes entirely: multi-GPU setups become mandatory, H100 units run $25,000 to $40,000 apiece, and configurations of 4 A100s or 2 H100s are common reference points for production deployments. Even the cheap end isn't cheap. Even a quantized 7B model on a handful of cloud instances carries meaningful fixed monthly costs just to stay available, before anyone sends it a real workload.

Aisuperior.com puts cloud pricing for H100s at $2.85 to $3.50 an hour on-demand across the major providers as of early 2026. Spot instances can dip to $0.80 an hour, and regional clouds offer $2.20 to $2.60 with weaker service guarantees attached. Then there's redundancy: production high-availability requirements mean running multiple nodes, which multiplies raw compute cost before the system processes one prompt from a real user.

The center of gravity has shifted, too. Inference, not training, now eats more than 80% of AI GPU spend, which means the hardware conversation is mostly about serving traffic, not building models. Hardware is diversifying in response. The H200, with 141 GB of HBM3e, is emerging as a high-memory option for the most demanding workloads. AMD's MI300X has become a more prominent alternative to the H100 for inference. Inference workloads themselves are drifting toward cheaper cards like the L40S and L4. None of this changes the underlying fact: model size and redundancy requirements set the hardware floor before any optimization work even starts. What piles on top of that floor is where things get genuinely expensive.

Why KV cache management is the most consequential infrastructure cost teams underestimate

The KV cache exists to stop a model from redoing work it's already done. Every time a transformer generates a token, it would otherwise have to recompute the key and value representations for every prior token in the sequence, which is wasteful and slow. Caching those representations, as described in arXiv 2603.20397, is one of the foundational tricks in transformer inference. It's also expensive in a way most teams don't budget for.

The expense scales with context length, and it scales linearly. Longer context means more GPU memory locked up per active request, full stop. A single Llama 3 70B request running at 128K context needs 42 GB of GPU memory just for the KV cache. On an 80 GB card, that leaves barely enough room for the model weights, and none at all for serving anyone else at the same time.

At a context length in the millions of tokens, the cache overtakes the model. At FP16, the KV cache runs around 135 GB, against roughly 140 GB for the weights themselves. The cache and the model are, at that point, roughly the same size, which is a strange thing to sit with: the thing that was supposed to be an optimization now costs almost as much memory as the model it's optimizing.

Agentic workloads make this worse, not better. Scaling active context from 8,000 tokens to 128,000 tokens balloons memory demand fast enough to trigger out-of-memory failures, or force quantization that quietly degrades the quality of what the model outputs. KV cache pressure is the single biggest lever on how many concurrent users a GPU can serve, which matters most for anyone doing cost planning. It sets utilization, and utilization sets cost per request. Everything downstream of this section is really about managing that one constraint.

Five KV cache optimization strategies and what they deliver

Arxiv 2603.20397 shows researchers have converged on five main directions for taming the KV cache: cache eviction, cache compression, hybrid memory schemes, new attention mechanisms, and combinations of the above. Some of these are architectural decisions baked in at training time, not knobs a deployment team can turn later.

Multi-Query Attention shares key and value representations across every attention head. Grouped-Query Attention shares them within groups of heads instead of across all of them. DeepSeek's Multi-Head Latent Attention re-parameterizes key and value into a lower-rank projection, shrinking the memory footprint by construction. YOCO takes a different approach entirely, letting the back half of a model's layers reuse KV state computed by the front half. None of these help a team running an existing open-weight model, since they're architectural choices made before the weights ever get published.

What does help right now is Paged Attention, which has become close to a universal default. It solves the requirement that KV memory sit in one contiguous block by scattering it across physical memory and tracking it with a page table, the same trick operating systems have used for decades. The cost is roughly 2 to 5% more compute per token; the payoff is effective memory utilization north of 95%. vLLM and TensorRT-LLM both ship it by default (SGLang instead ships RadixAttention as its default).

Prefix caching is arguably the highest-leverage trick available to a team that isn't training its own model. It hashes the shared prefix of incoming prompts and reuses the KV state computed for that prefix, rather than recomputing it every time. vLLM's prefix cache and SGLang's RadixAttention both implement versions of this, and on workloads with a high cache-hit rate, it delivers 85 to 95% in cost savings. Stacking all five strategy families together can drop long-context inference cost by a wide margin, several times over at the low end and dramatically more at the high end, depending heavily on the shape of the workload.

Even with every optimization applied, GPU memory is finite. At some point, KV state has to live somewhere other than the GPU, and that turns into a storage engineering problem.

The storage tier that production inference now requires

Every tier below the GPU is dramatically slower, and the gap sets the terms for what's actually possible. An H100 SXM5 moves data through HBM at roughly 3.35 TB/s. Bandwidth falls to around 63 GB/s on CPU DRAM over PCIe 5.0 x16. Local NVMe drives run about 7 GB/s. Networked storage sits further out still, serving as the capacity tier for anything persistent or shared across nodes.

That gap is why tiered KV caching has become a real production pattern rather than a nice-to-have: hot state in VRAM, warm blocks in CPU DRAM, cold blocks on NVMe, and persistent or shared state on the network tier. Software like LMCache manages this hierarchy underneath vLLM and NVIDIA Dynamo, and reported latency reductions on cache-hit-heavy workloads range from several times over to an order of magnitude.

NVIDIA announced its own answer at CES 2026: the Inference Context Memory Storage Platform, or ICMSP, which standardizes offloading KV cache to NVMe SSDs and extends GPU cache capacity out into NVMe storage built with NVIDIA's storage partners. NVIDIA claims substantially better power efficiency and substantially higher tokens-per-second from the platform. Storage partners report strong results running on it: VAST Data reported roughly 10× faster prefill times using ICMSP with all-NVMe storage, which translates directly into higher concurrency per GPU. The transport layer underneath ICMSP, called NIXL, is designed to support multiple interconnect types, giving it flexibility across different cluster topologies.

NVIDIA Dynamo extends the idea into a full four-tier scheme: GPU HBM, CPU DRAM, local SSD, and networked storage, with KV-aware routing that sends incoming requests to whichever node already holds the relevant cached state. It plugs into vLLM and compatible inference frameworks. Above the node level, additional context memory tiers can bridge local node storage and shared network storage.

One detail gets skipped constantly and shouldn't: KV traffic is continuous and write-heavy, which means it needs datacenter-grade NVMe drives built for sustained endurance. Drives not rated for sustained write workloads degrade under that pattern. Teams that try to save money by cutting the storage tier find that out the hard way, usually in production. The industry has noticed the shift, too: FMS 2026, the event formerly known as Flash Memory Summit and now rebranded Future of Memory and Storage, played out less like a niche storage conference and more like an AI memory infrastructure event, with NVIDIA, Samsung, SK Hynix, Kioxia, SanDisk, and Micron all presenting NAND and related memory as core components of the AI accelerator stack. That's a structural signal, not a passing trend. The bandwidth gap between HBM and NVMe spans several orders of magnitude, which means decisions about how KV blocks get placed, evicted, and fetched aren't a commodity storage question. They determine whether a system is fast, affordable, or workable at all.

Engineering labor: the cost that grows with system complexity, not just with token volume

Hardware and storage are visible costs. Labor is the one that hides. Someone has to deploy the system, watch it, patch it, update the models running on it, and get paged when it breaks, and that someone's time is the least-budgeted line item in most self-hosting plans.

Sitepoint.com puts the time to get a deployment production-ready in the first place, meaning Kubernetes clusters configured, load balancers set up, CI/CD pipelines built, monitoring wired in, at about 2 to 4 weeks of senior DevOps time. That's before ongoing maintenance even starts. After launch, expect 10 to 20 hours a month for security patches, dependency bumps, scaling tweaks, and general troubleshooting. Truefoundry.com puts twenty hours a month at a senior engineer's fully-loaded rate at about $1,730. Sitepoint's estimate for allocating 20 to 30% of a senior engineer's time to ongoing operations is $3,000 to $6,000 a month, which is a wide enough range to suggest the number depends heavily on how complicated the deployment actually is.

Headcount scales with ambition. A minimal internal deployment needs 3 to 4 engineers. Anything customer-facing needs 7 to 10. Enterprise scale runs 15 or more specialized people. ML engineers command $150,000 to $250,000 a year; MLOps engineers run $140,000 to $230,000. And roughly 60% of the actual engineering effort on these systems goes into what's sometimes called glue code: wiring the model into databases, authentication, and whatever interface end users touch. That percentage doesn't shrink as the system matures. If anything, more integrations mean more glue.

Even tools marketed as free carry a hidden bill. LiteLLM, the open-source LLM proxy, costs nothing to self-host, but running it in production adds $200 to $500 a month in supporting infrastructure on top of the DevOps time required to keep it healthy. Truefoundry.com puts its enterprise tier at $250 a month climbing to $30,000 a year, which makes the point cleanly: even lightweight open-source tooling carries a total cost of ownership well past its license price. Incident response costs are absent from a pre-launch budget entirely. A 2 a.m. outage gets handled by whoever's on the internal on-call rotation, not by a vendor's support desk, and that cost simply doesn't exist in most planning spreadsheets until it happens.

With hardware, storage, and labor mapped out, the real question is where the math actually favors self-hosting, and where it doesn't.

Where self-hosting breaks even

Most organizations hit cost parity with commercial APIs somewhere in the hundreds of millions of tokens a month. Factoring in the full weight of engineering overhead for a 70B-class model drops the practical break-even point to a much lower monthly token volume, with FP8 quantization pushing cost-per-million-tokens down to about $0.95 to $1.10. Below that volume, the fixed costs outlined above simply don't have enough traffic to spread across.

The volume trap makes this harder to plan around than it sounds. A pilot running a modest volume of tokens a day can hit a production volume dozens of times larger, a jump that can carry a team past break-even without anyone making a deliberate call to cross that line. Volume alone doesn't decide the outcome, either: latency requirements, data residency rules, how long a context window needs to be, and whether the engineering team has spare capacity all shift where the real break-even point sits.

Any honest model has to include hardware capital or hourly cloud rates plus the redundancy multiplier, the storage tier with its endurance ratings and networked capacity for shared KV state, labor covering both initial deployment and ongoing maintenance and incident response, and the TCO of tooling that looks free until DevOps time gets added in. If any one of those is skipped, the projection is fiction.

For teams that do clear the threshold, the return comes from the same two levers covered above: KV cache optimization, where prefix caching alone can save 85 to 95% on cache-hit traffic and combined strategies can cut cost by a wide multiple, and storage architecture, where tiered KV management delivers latency improvements ranging from several times over to an order of magnitude on cache-hit workloads. Hardware and headcount get budgeted because they're visible and someone has to sign off on the invoice. Storage throughput and KV cache design get skipped because they're invisible until the system falls over, and that's exactly where most teams leave performance and money sitting on the table. Run consumer SSDs past their write endurance, skip the DWPD rating, ignore tiered offloading entirely, and the cost appears later in throttling, out-of-memory crashes, and degraded output, not as a line item. It appears as throttling, out-of-memory crashes, and degraded output, right when traffic is at its highest.