CapEx vs. OpEx Models for AI Infrastructure at Startups vs. Enterprises

The CapEx versus OpEx question in AI infrastructure is an engineering decision. It's an engineering decision that shows up in GPU memory allocation, KV cache placement, and storage bandwidth, and getting it wrong means either paying for idle H100s or hitting a throughput wall you can't push past. The cloud-era version of this debate was mostly about compute elasticity: rent when demand is spiky, buy when it's steady. AI infrastructure adds a harder layer on top of that, because memory residency and cache persistence don't bend to simple consumption pricing the way a stateless web server does.

The macro spending context that shapes every infrastructure negotiation in 2026

Start with the number that sets the frame for everything else. The five largest US cloud and AI infrastructure providers have collectively committed somewhere between $660 billion and $690 billion in capital expenditure for 2026. Amazon is putting in roughly $200 billion. Alphabet's commitment is close in size to Amazon's, landing in a comparably high range. Meta's commitment is somewhat below Alphabet's range, Microsoft's is comparable to Meta's, and Oracle has committed a much smaller sum by comparison. These aren't annual totals padded over years; in a single quarter of 2025, Microsoft deployed $34.9 billion, AWS spent $28.3 billion, and Google put in $23.95 billion. That's the kind of capital velocity that sits well outside what any enterprise, let alone a startup, can match.

Goldman Sachs' baseline aggregates put cumulative AI capital spend between 2026 and 2031 at roughly $7.6 trillion. That commitment is locked in now, and it drives the buildout of the physical infrastructure that supports AI, the GPUs, the NVMe arrays, the interconnects, the colocation footprint, being built at a scale that does two things at once. It gives hyperscalers real pricing power over rented capacity, and it seeds a parallel market of owned infrastructure for anyone who can get financing on reasonable terms.

None of this actually settles whether renting that capacity makes sense for a given workload. The macro numbers describe scale, not fit. Whether OpEx or CapEx wins for a specific team depends on utilization patterns, memory footprint, and storage throughput, questions the trillion-dollar headline can't answer for you.

GPU depreciation and obsolescence risk as the hidden CapEx penalty

Buying GPUs outright carries a cost that doesn't appear on the invoice: the clock starts ticking the day the successor chip launches. Prior GPU generations, V100 to A100 to H100, lost 40 to 60% of their resale value within 18 to 24 months of the next generation shipping. Economic depreciation on these chips is front-loaded, 30 to 40% in year one alone, and the useful economic life runs 3 to 4 years against the 5 to 7 years a traditional server holds up. An H100 cluster bought in early 2023 is, by 2026, sitting two generations behind, still on the books but no longer representing current-generation performance per dollar.

This hits pure CapEx buyers asymmetrically. Renters absorb the same obsolescence cost implicitly, baked into whatever the vendor charges, but they never carry the balance sheet exposure. The asset risk gets socialized across every customer instead of concentrated on one buyer's depreciation schedule.

A handful of financing structures have emerged specifically to blunt this. Reporting from The Information and CNBC laid out OpenAI's plan to lease NVIDIA GPUs under five-year arrangements instead of buying them outright, a structure that could cut hardware costs by 10 to 15%. CoreWeave raised $2.3 billion by pledging its H100 fleet as loan collateral. Lambda ran a $1.5 billion sale-leaseback. All three treat GPUs less like depreciating infrastructure and more like tradeable financial assets, which is a genuinely different posture than the buy-it-and-own-it model most enterprise IT procurement was built around.

This obsolescence pressure is concentrated at the compute tier. NVMe and networked storage don't face a generational cliff the way GPUs do, so the useful life of a storage investment runs longer and the depreciation math looks nothing like the GPU side. That asymmetry alone argues for owning different tiers of the stack differently, not applying one rule about buying versus renting across the whole system.

There's also a new accounting wrinkle. FASB's ASU 2025-06, issued in late 2025, introduced what amounts to a "significant development uncertainty" hurdle: novel or unproven AI projects may have to expense costs immediately rather than capitalize them over time, until that uncertainty resolves. For any organization hoping to smooth AI infrastructure spend across several fiscal years the way it would with a data center buildout, this closes off some of that flexibility.

The utilization threshold that determines when owned infrastructure wins

Everything in this piece organizes around one number. Owned infrastructure beats renting from the cloud once sustained utilization crosses roughly 60 to 70%. Fall below 50%, and leased or cloud capacity wins on cost every time. That threshold is simple to state and brutally hard to hit with AI workloads, because inference load doesn't behave like a steady batch job.

A conversational assistant has a fairly predictable token count per exchange. Agentic workloads don't. A single user instruction can fan out into dozens or even hundreds of separate model calls, chained across tool use, retrieval steps, and sub-agent delegation, which makes projecting sustained utilization months out an exercise in guesswork rather than forecasting.

The cost consequence is visible in the numbers. An analysis from Andreessen Horowitz found several AI-native startups spending 60 to 80 cents of every revenue dollar on model inference alone, a ratio that doesn't compress with scale the way traditional SaaS gross margins do. That's the trap: high OpEx inference bills paired with low utilization on any owned hardware sitting alongside them, the worst of both models at once.

The 60-70% rule also flattens something important: GPU utilization and storage utilization aren't the same metric and shouldn't be graded on the same curve. NVMe arrays and networked storage tiers sustain high utilization across mixed workloads, training runs, inference serving, checkpointing, in ways a GPU cluster simply can't, because storage demand is smoother and less bursty than compute demand. And in long-context serving specifically, storage throughput can become the binding constraint before GPU utilization ever does. KV cache offloading from GPU memory to CPU DRAM adds 10 to 50 milliseconds per cache retrieval; offloading further out to NVMe adds additional latency on top of that. Those numbers determine throughput and latency in practice more directly than whatever percentage the GPUs are sitting at.

So the 60-70% threshold is a fine rule of thumb for deciding whether to buy GPUs. It understates, pretty badly, the case for owning the storage layer underneath them, where utilization runs more stable and the performance gap between owned NVMe-over-Fabrics and general cloud object storage is measurable.

Startup-Scale AI Infrastructure and the OpEx Ceiling

Startups default to OpEx inference for reasons that make sense on day one and stop making sense somewhere around month eighteen. There's no capital available for a GPU cluster at that stage, the workload is genuinely too uncertain to plan around, and nobody wants to commit to reserved capacity when agentic workloads might fan a single request into a hundred model calls next quarter. Renting by the token is the only rational choice early on.

The ceiling shows up quietly. Token costs are variable, but they aren't elastic in the way SaaS cost of goods sold is supposed to be; they don't compress as volume grows the way a per-seat software cost does. Longer context windows mean bigger KV caches per request, and that means more memory consumed per inference call, raising the real cost per token in a way that never appears as a line item on the invoice. It appears instead as a throughput ceiling, a latency spike, or a bill that doesn't match the usage graph anyone expected.

The real inflection point comes when a startup moves off managed inference APIs and onto self-hosted open-source models, Llama, Qwen, Mixtral, whatever fits the use case. The question of buying versus renting stops being abstract at that moment, because KV cache management, NVMe provisioning, and GPU memory tiering are no longer someone else's problem hidden behind an API endpoint.

And the first thing most teams discover at that transition is how wasteful naive KV cache allocation actually is. Before an optimization called PagedAttention became standard, LLM inference systems were wasting 60 to 80% of allocated KV cache memory through fragmentation and over-allocation. For a startup that just took on the cost of self-hosting, that waste is the first place engineering decisions translate directly, dollar for dollar, into infrastructure spend.

What PagedAttention and KV cache optimization change about the ownership calculus

PagedAttention, introduced by the vLLM project, cut that KV cache waste from 60 to 80% down to a small fraction and delivered several times the throughput gains in the process, a change to the underlying economics of running inference. That's a change to the underlying economics of running inference, because the same GPU capacity now serves several times as many concurrent requests as it did before. It's a change to the underlying economics of running inference, because the same GPU capacity now serves several times as many concurrent requests as it did before.

That throughput gain matters directly for the utilization threshold discussed above. If PagedAttention lets a fixed GPU fleet serve more concurrent traffic, it pushes utilization closer to the 60 to 70% band where owning the hardware starts to beat renting it. vLLM's V1 engine, the default since 2025 and now iterating past version 0.25 according to the project's GitHub releases, has added chunked prefill, better scheduler logic, and multi-modal support on top of that. The open-source inference stack has gotten sophisticated enough that raw performance is no longer a good reason to default to a managed API.

What PagedAttention doesn't solve is the underlying size problem. A 70B model at 8K context needs roughly 20 GB of KV cache per request, which balloons to around 640 GB for a batch of 32. That number doesn't shrink because fragmentation is fixed; it shrinks only through more HBM (a CapEx decision), offloading to NVMe (a storage infrastructure decision), or quantization (a compression research problem).

That third path is where a lot of current work is concentrated. KVzip applies query-agnostic compression to shrink cache footprint. KIVI uses 2-bit asymmetric quantization, published at ICML 2024. Google's TurboQuant targets KV memory compression down to as little as 3.5 bits per value. Each of these reduces the amount of HBM a long-context serving system actually needs, which changes how much owned GPU memory a team has to buy.

Investing engineering time in KV cache compression is a real substitute for buying more hardware, but only if the tier you're offloading to, NVMe or CPU DRAM, can absorb that data without blowing the latency budget. Compression buys you headroom on the CapEx side only if the storage side can keep up.

The GPU memory hierarchy as the real CapEx decision surface

The actual decision isn't "buy GPUs or rent them." It's which layer of the memory hierarchy to own. That hierarchy runs from HBM sitting directly on the GPU (fastest, most expensive) down through CPU DRAM, local NVMe, networked NVMe-over-Fabrics, and finally cloud object storage, and each tier has its own cost per gigabyte, its own latency, and its own ownership logic.

The current generation of flagship hardware sets the anchor points. NVIDIA's B200 carries 180 GB of HBM3e per GPU, 1,440 GB across the eight GPUs in a DGX B200 system. The H200 carries 141 GB of HBM3e, the H100 carries 80 GB of HBM3. Those figures are what any team modeling its own inference infrastructure has to build its capacity plan around.

HBM is expensive in a way that resists brute-force scaling. It carries a substantially higher cost per gigabyte than ordinary DDR memory, manufacturing capacity for it is limited, and its growth rate lags behind how fast model sizes and context lengths are expanding. Buying more HBM cannot be the whole answer, because supply and cost don't move fast enough to keep pace with KV cache growth.

Tiering is the practical answer, and it comes with a quantifiable tax. Keeping only the "hot" KV tokens resident in HBM and pushing intermediate reasoning tokens down to CPU DRAM carries roughly 5 to 7% transfer overhead, a cost that's small enough to trade against the CapEx saved by not buying more GPU memory. A newer layer is starting to sit in between HBM and NVMe: CXL-attached memory. At GTC 2026, Penguin announced a CXL-based memory server aimed at delivering lower latency and higher throughput to GPU clusters than reaching all the way to NVMe would allow, expanding the addressable memory pool without adding more GPU cards.

Framed this way, the CapEx decision gets more specific and more useful. It's which of these five tiers a team owns outright, which it rents, and which it abstracts away entirely, not "buy GPUs, yes or no."" It's which of these five tiers a team owns outright, which it rents, and which it abstracts away entirely, and that's a decision that can actually be defended in front of a CFO.

NVMe and networked storage as the tier where CapEx ownership pays most reliably

Storage is where owning infrastructure pays off with the fewest caveats, and the architecture explains why. AI storage systems increasingly use NVMe-over-Fabrics and disaggregated designs that scale bandwidth independently of raw capacity, built specifically for the high-concurrency demands of large models and distributed training. That's a categorically different animal from general-purpose cloud object storage, which was built for durability and cost per gigabyte, not for feeding hundreds of GPUs simultaneously.

GPUDirect Storage is the clearest expression of that difference. It moves data straight from storage into GPU memory, skipping the CPU and system memory. Paired with InfiniBand or RoCE networking and a compatible backend, a multi-GPU cluster can pull substantial aggregate read bandwidth for dataset prefetching. It requires RDMA-capable networking, InfiniBand or RoCE with the right NICs, rather than standard TCP/Ethernet.

The throughput gains attributed to GDS come specifically from removing CPU I/O stalls in the data-loading pipeline that feeds GPUs during training. That benefit applies specifically to workloads bound by data loading, not universally across every training or inference job. Where it does apply, though, it's a performance gap that cloud object storage architecturally cannot close, because the path it uses routes data through the CPU and adds a hop that GDS was built to remove.

NVMe-over-Fabrics deployments continue to expand as more organizations extend high-speed storage fabrics to match the concurrency demands of modern training clusters. Storage is the layer of this whole stack where the ownership case is least ambiguous: the hardware doesn't face the same obsolescence cliff as GPUs, utilization runs steadier across mixed workloads, and the performance ceiling of owned, direct-connected NVMe is one that general-purpose rented storage isn't built to reach.