On-Premises GPU Clusters vs. Cloud Spot Instances for LLM Training
Ownership breaks even when sustained utilization covers staff, hardware, and infrastructure costs.
On-premises GPU clusters versus cloud spot instances for LLM training is a utilization decision. Ownership pays only when sustained productive use covers the full cost of staff, hardware, storage, and networking. Spot wins when elasticity and lower rental prices outweigh interruption costs, which depend heavily on checkpoint throughput and recovery.
Why the on-prem vs. cloud spot decision is really a utilization threshold problem
Hardware specifications and per-hour prices are the wrong starting frame because neither reveals how much useful training a deployment will deliver. What matters most is sustained utilization after time lost to idle periods, maintenance, communication waits, and checkpoint writes. Installed capacity is not the same as productive capacity. A cluster that looks inexpensive when fully occupied can lose its advantage when the training schedule leaves gaps.
Slow storage reduces productive GPU time during checkpointing, while constrained networking stretches multi-node training. Both make a workload cost more to finish even while the GPUs are allocated to it. Because it takes time to change storage and fabric designs after purchase, a team that overestimates utilization can get stuck with high costs and low throughput.
What the true total cost of ownership looks like for an on-prem H100 cluster
Staff is the largest line item in the on-prem H100 cost model described here. Hardware depreciation comes second, amortized over three years. That ordering changes the purchase calculation substantially. Budgeting for servers while treating infrastructure engineering as spare capacity understates ownership cost and pushes the apparent break-even below its true level.
Power is the next distinct component, with regional rates producing a substantial spread between US Pacific Northwest industrial supply and European commercial supply. Cooling adds another large share to the electricity bill. Liquid cooling cuts that cost but needs spending on infrastructure of its own. Neither expense can be inferred from the GPU purchase price alone.
InfiniBand switches and cabling form a discrete capital expense, while NVMe capacity for datasets and checkpoints grows with model size and checkpoint frequency. Financing adds another cost to the hardware total. Teams that finance equipment therefore need to work out break-even differently from teams that buy outright, even with the same servers and workloads.
Where the utilization break-even actually sits against cloud on-demand and spot pricing
The annual cloud comparison follows a simple formula. The cost of cloud for a year is price per GPU-hour × GPU count × hours per year × utilization rate. Break-even utilization is annual on-prem TCO divided by the product of the first three terms. The denominator changes with the provider and purchasing option. There is consequently no universal utilization threshold that separates ownership from rental.
For a worked comparison, a team would enter its annual ownership budget, then multiply its provider’s current on-demand GPU-hour price by the proposed GPU count and annual hours. Dividing the cost of ownership by what rental would cost at full utilization gives the on-demand threshold. Doing the same calculation with the current spot rate gives the spot threshold before costs from interruptions. When rental rates fall, utilization must be higher for ownership to pay.
The supplied cost components do not establish a defensible numerical threshold without actual staffing costs and live rental prices. A percentage presented without those inputs would be false precision. Hyperscaler on-demand pricing can make ownership look attractive where specialist-cloud on-demand does not, and spot can shift the comparison further toward rental. The practical requirement is to rerun the formula against the provider actually under consideration, with AWS on-demand serving as a provider-specific benchmark.
How spot instance preemption risk changes the calculus for multi-week training runs
Spot pricing is only the starting cost. For multi-week training, the comparison must also account for interrupted work, checkpoint overhead, and recovery time. AWS GPU spot reclamation presented a material interruption risk in early 2026, with exposure varying by region. An owned cluster avoids spot reclamation, but it still has downtime for maintenance and other operational work.
Checkpoint frequency determines how much progress is exposed between saves, but faster saving is not free when writes stall training. More frequent checkpoints reduce restart loss while increasing GPU idle time if storage is slow. Even at a low rental rate, enough time spent saving or repeating work can wipe out the savings.
GPU Direct Storage provides a concrete mitigation by allowing the GPU’s DMA engine to write directly to NVMe, bypassing a CPU-staged path. The supplied example puts a 140 GB checkpoint below 40 seconds with GDS. That makes frequent saving more viable. For workloads where a restart can cost days of progress, though, ownership’s preemption-free execution has value beyond its effect on price.
Checkpoint throughput and NVMe storage as the hidden bottleneck in both environments
A 140 GB BF16 checkpoint makes storage performance tangible. Its write latency consumes time that could otherwise support training, regardless of whether the GPUs are owned or rented. Many teams do not measure this utilization tax. On spot, the tax becomes more prominent because frequent saves are part of the strategy for limiting restart loss.
The architectural distinction is between the CPU-staged write path and direct GPU-to-NVMe transfer. The GDS result above depends on that path being available in the chosen environment. Buying faster GPUs does not remove a checkpoint bottleneck. Storage throughput must support the save frequency required by the workload’s recovery policy.
Read throughput imposes a related constraint when feeding training jobs. MLPerf Storage v3.0’s September 2026 supplemental reports production-scale training reads reaching tens of GiB/s over RDMA-connected parallel filesystems. Properly configured InfiniBand/RDMA storage can approach that range, while cloud network-file-system-backed storage typically cannot.
How InfiniBand vs. cloud Ethernet changes training throughput at multi-node scale
A 2× training-time gap is large enough to overturn a GPU-hour price comparison, with its relevance to cloud Ethernet depending on model size and communication volume. For large multi-node jobs, the fabric can become the binding constraint before GPU generation does. For single-node fine-tuning, that inter-node advantage is irrelevant.
The supplied NCCL all-reduce comparison on 8×H100 SXM5 reports roughly 350 GB/s effective bus bandwidth with InfiniBand NDR 400G, against 270–290 GB/s with 400GbE RoCEv2. InfiniBand combines low-latency communication with lossless RDMA. With that design, data can move directly between GPU memory on separate nodes without going through the CPU or kernel.
MLCommons MLPerf Training 2024 results show higher tokens per second for InfiniBand NDR clusters than equivalent RoCEv2 clusters at the same GPU count, with a smaller gap for smaller models. Above 70B parameters, large all-reduce volumes can make this distinction decisive. RDMA can also extend the fabric’s role into storage through NVMe-oF, making remote flash access part of the same deliberate fabric design.
Precision formats, lossless compression, and what they mean for storage and network traffic during training
Precision changes the volume of data that storage and networking must handle. FP32 serves master weights and gradient accumulation, BF16 serves forward and backward passes in most production runs, and FP8 serves Transformer Engine-optimized workloads. These roles need separate accounting. A format used for one part of training does not by itself describe every stored or communicated tensor.
For the model-weight payload, a 70B model occupies 140 GB in BF16 and 70 GB in FP8. The smaller representation fits on a single B200 instead of two H100s, changing GPU memory requirements as well as storage volume. The full training-state calculation must still include the formats used for the other parts. Otherwise, smaller weights can make the checkpoint budget look smaller than it needs to be.
Lossless compression is another way to cut traffic while keeping the represented values the same. BF16 has 1 sign bit, 8 exponent bits, and 7 mantissa bits, with non-uniform exponents providing the main compression opportunity while sign and mantissa bits approach uniform distribution. ZipCCL examines exploiting that structure in AllReduce communication. For BF16 workloads, native support for compressed checkpoints and collectives can partly narrow the storage and communication gap, but the benefit depends on support in the actual stack.
Workload profiles that clearly favor on-prem and those that clearly favor cloud spot
Clear on-prem cases include sustained multi-node pre-training above 70B parameters where all-reduce traffic makes InfiniBand NDR a throughput requirement. Production pipelines that cannot tolerate spot reclamation also favor ownership. Data locality strengthens the case further. When large proprietary corpora already reside on-premises and move daily, transferring them for cloud training adds a recurring cost that GPU-hour comparisons omit.
Clear cloud spot cases are preemption-tolerant workloads whose measured utilization is too low for ownership to pay and whose checkpoint path keeps the cost of recovery low. With smaller models and single-node fine-tuning, inter-node InfiniBand has no advantage in that comparison. Cloud elasticity helps most when there are long gaps in demand. Running on a single node, though, does not make spot a good fit if the team cannot accept the losses from interruptions.
The genuinely ambiguous middle consists chiefly of teams that cannot yet establish their sustained utilization. Their uncertainty is an input problem. The recommended response is to run in cloud for six months, measure actual use, and revisit ownership with that record.
The infrastructure constraints that make or break each path at training scale
For on-premises training pipelines, high-throughput NVMe in RAID 0 is the stated performance baseline, with PCIe Gen 4 the minimum for heavy ML workloads. The access path needs just as much care when storage is remote. The same MLPerf supplemental reports over 21 GiB/s from a shared remote parallel filesystem using two independent 400 Gb/s NDR InfiniBand/RDMA links. That bandwidth depends on how the system is built, and having NVMe alone does not provide it.
NVMe-oF over RDMA enables direct GPU reads from remote NVMe with sub-millisecond latency. Automated tiering keeps hot training datasets on NVMe while moving cold logs to object storage. Placement is part of throughput engineering. A fast tier cannot support the pipeline if the active dataset is elsewhere when the job needs it.
Data locality should therefore enter the comparison before compute selection. Daily movement of 2 TB creates a recurring transfer requirement, but the applicable charge depends on the data path and billing direction. AWS egress rates should not automatically be applied to data entering AWS from an on-premises lake. The cloud TCO needs the charges that actually apply to the planned movement.
Operational work also consumes the capacity that the financial model expects to amortize. Maintenance, driver debugging, and fabric reconfiguration reduce effective utilization when they take a cluster out of service. Those hours belong in the calculation. Storage topology, placement, networking, and operational constraints are manageable, but their management must be deliberate and its cost must appear in the TCO comparison.
A concrete decision framework for making the call before you commit capital
First, measure utilization under production-equivalent load before working out how much ownership could save. Run workloads in cloud for one to three months and instrument GPU utilization. Record checkpoint stalls and communication waits as part of that exercise. If the schedule is still too uncertain to decide whether to buy, keep gathering evidence for the six months described above.
Second, build the annual ownership budget from staff, three-year hardware depreciation, power, cooling, networking, NVMe storage, and financing where applicable. Compare it with the cloud formula using current prices from the intended provider. Calculate on-demand and spot cases separately. For spot, add checkpoint overhead and interrupted work to the quoted rental rate to work out the cost of finished training.
Third, identify the workload’s binding constraint before accepting the cheaper spreadsheet result. A job that requires NDR InfiniBand throughput needs a deployment that supplies it. A checkpoint-bound job requires evaluation of GDS-capable NVMe in cloud against an on-prem NVMe-oF design. If the corpus is on-premises, the applicable movement charges belong in cloud TCO before the comparison closes.
Finally, budget operational ownership explicitly and revise utilization to reflect downtime. The staffing model here calls for one full-time infrastructure engineer per small group of GPUs. That expense belongs beside the capital budget. Storage and interconnect performance belong inside the utilization calculation itself, because they determine how much paid GPU time becomes useful training.
