Checkpoint Storage Cost During LLM Training Runs
Optimizer state, not model weights, drives checkpoint costs during training.
Checkpoint storage during LLM training gets filed under overhead, a line item below the GPU bill that nobody scrutinizes closely. Filing checkpoint storage under overhead as a line item nobody scrutinizes closely is a mistake. Checkpoint size, frequency, and placement are engineering decisions that compound over a training run, and teams that treat them as fixed defaults end up paying for it twice: once in storage, once in idle GPU time they never notice until the bill arrives. This piece covers the mechanics behind that cost, and where the actual decisions sit.
What a training checkpoint contains and why the optimizer state dominates its size
A checkpoint holds everything needed to pick a training job back up exactly where it left off if a node dies or gets killed. That means model weights, but it also means optimizer states (for Adam, a first moment and a second moment tensor per parameter, plus an FP32 master copy of the weights), and related training metadata needed to restore the exact training state.
Mixed-precision training is where the size math turns ugly, and it's where most people misjudge what they're actually storing. Weights in FP16 cost 2 bytes per parameter. Adam's optimizer state, kept in FP32 for numerical stability, adds another 12 bytes per parameter on top. An arXiv paper on checkpoint sizing (TierCheck) adds the two together and puts a full checkpoint at roughly 14 bytes per parameter. The weights are almost a rounding error next to that. The optimizer state is the checkpoint, and any storage plan built around "model size" instead of "checkpoint size" starts from the wrong number.
AWS's SageMaker HyperPod documentation gives real figures: a Llama 3 70B checkpoint with optimizer state runs around 521 GB. DeepSeek-R1 lands closer to 5 TB per checkpoint. MLCommons' MLPerf Storage v2.0 benchmarking pushes the math to a trillion-parameter model and finds the optimizer state alone at 13.2 TB out of a roughly 15 TB total. The 14-bytes-per-parameter rule lets a team estimate checkpoint size before a run starts. The sane planning number is 3 to 5 times the final model's size in active checkpoint storage, for every concurrent run in flight.
How checkpoint frequency and run length multiply storage consumption
Frequency is where the arithmetic gets alarming. Most large model training checkpoints every 500 to 2,000 steps, and each write pushes somewhere between 350 and 500 GB of state out of GPU memory and onto persistent storage. That's per checkpoint. Multiplying that by the number of checkpoints in a multi-week run makes the totals stop looking like rounding errors.
MLCommons modeled a trillion-parameter model writing a 15 TB checkpoint every 90 seconds, a pace that produces over 14 petabytes of writes in a single day. At cloud storage pricing of $0.05 to $0.12 per GB per month, retaining checkpoints at that scale is a bill that keeps climbing for as long as the run continues and nobody deletes anything.
Retention policy is where most of the waste actually hides, more than write frequency itself. Teams that keep every checkpoint from every run, every experiment, every version, are paying for storage that has nothing to do with model performance and everything to do with nobody owning the delete button.
Checkpoint cadence usually gets set once, early in a run, based on a failure-risk calculation: how many hours of compute the team is willing to lose if a node crashes. That early setting is the deeper problem, because it almost never gets revisited once the run stabilizes. A cadence that made sense in week one, when crashes were frequent and the blast radius of losing progress was small, keeps running unchanged into week four, when it no longer matches the actual risk. Versioning and experiment logging stack more volume on top of the training checkpoints themselves, and most initial storage budgets never account for it.
The GPU idle cost hidden inside synchronous checkpointing
Synchronous checkpointing stops the world. Every GPU across the cluster sits idle while training state writes to storage, and nothing resumes until the write finishes. An arXiv paper on agentic AI infrastructure puts that pause at 30 to 60 seconds per save for a 70B model with a checkpoint around 840 GB including optimizer state.
That sounds small until you multiply it out. At cloud H100 pricing of $2 to $10 per GPU hour, a week-long job running 1,000 checkpoints burns through 67 to 80 hours of pure GPU idle time. At an on-demand rate of $4.06 per hour per H100 across an 8-GPU node, that idle time alone costs over $2,100, before counting the training progress that never got made.
The wasted time gets misattributed almost every time. It appears not on the storage invoice but in GPU usage on the compute bill, so teams that scrutinize storage costs while leaving checkpoint cadence on a framework default are reading the wrong ledger. Checkpoint strategy is an infrastructure decision. Get it wrong and it taxes both budgets at once.
Async checkpointing for that same 70B model takes under a second, non-blocking, and incremental or delta saves move only 5 to 20 GB instead of the full 840. Thirty to sixty seconds of blocked GPUs against under a second of no blocking at all is not a marginal difference. It's close to an order-of-magnitude gap, and over a multi-week run that gap either stays invisible or starts eating real money.
Asynchronous and tiered checkpointing as the primary architectural response
Async checkpointing copies state to node-local storage first, releases the GPUs immediately, and lets the slower write to shared or global storage happen in the background without blocking anything. Paired with a tiered storage architecture, this is the correct default, not one option among several: a fast tier of node-local NVMe absorbs frequent writes at full speed, while a slower shared tier, usually a parallel file system or object store, gets written less often to guard against node-level failure.
Production data backs this up. Storage infrastructure analysis covering 40 production training runs found an 800-billion-parameter job checkpointing every 40 minutes with a median checkpoint duration of just 3.6 minutes, roughly 9% overlap with training time. Nearly all 40 runs studied kept that overlap under 10%. Checkpoint overlap as a share of total training time is the metric that actually matters here, not raw throughput in gigabytes per second, and 10% is the number to design toward.
Google's AI Hypercomputer team reported a 6.59% increase in ML Goodput from multi-tier checkpointing on a 35,000-chip TPU v5p workload, published in a Google Cloud blog post. That's a concrete figure for what this architecture is worth at real scale. AWS validated similar managed tiered checkpointing across SageMaker HyperPod clusters ranging from a few hundred GPUs up past 15,000.
There's a second payoff that doesn't get enough credit. Spot and preemptible GPU instances cut training costs by 60 to 70% against on-demand pricing, but they only work if checkpointing is fast and reliable enough to survive frequent preemption without losing hours of work. Checkpoint strategy and GPU procurement strategy are coupled decisions: cheap compute is only cheap if the checkpointing behind it holds up, because unreliable checkpointing forces retries that erase the savings from cheap instances. The failure math favors this approach anyway. Whole-node failures are rare; subcomponent issues, a flapped network link, a GPU throwing ECC errors, are common, and local checkpoints handle those fine on their own. The shared, slower tier only has to protect against the rarer, catastrophic event.
Sizing the storage layer: bandwidth requirements and the NVMe-to-parallel-filesystem stack
Once the tiering decision is made, the next question is how much bandwidth the shared tier needs. The formula: checkpoint bandwidth equals checkpoint size times frequency, divided by acceptable overlap times total training time. For a trillion-parameter model with a 15 TB checkpoint, MLCommons found the required checkpoint write bandwidth reaches 3.6 TB/s at the cluster level, a number that underscores why the shared storage tier's sustained throughput requirement is substantial.
No single storage system hits that number directly. Async checkpointing through node-local NVMe substantially cuts the bandwidth demand on shared storage, and that's what turns an impossible target into an achievable one. Production drain rates run 50 to 200 GB per second across observed training jobs, moving data from local NVMe to the shared tier, with no strong pattern tying that rate to model size.
Most AI training servers ship with 4 to 8 NVMe drives per node, and the aggregate bandwidth comes from writing across all of them in parallel, not from any single drive's rated speed. On the shared tier, two categories cover the field. Parallel file systems like Lustre or IBM's Spectrum Scale use POSIX semantics and get their throughput by striping data across many storage targets (a 512-GPU cluster needs 400 to 600 GB/s sustained during data loading alone). Cloud object storage is cheap for cold retention but was never built for frequent, high-bandwidth writes. Higher-performance managed storage tiers command a price per TB per month several times higher than standard object storage, depending on configuration.
Trying to checkpoint a trillion-parameter model synchronously means writing 15 TB in under 5 seconds, an average of 3.6 TB per second across the cluster. That number alone settles the argument for async plus local NVMe. At that scale, it is the only path that works. It's the only path that works.
Multi-tier TCO: where the cost difference between storage architectures lands
The simplest architecture is to buy high-bandwidth NVMe for everything. It's tempting because it removes a design decision, and it's the wrong call for almost every team, because it's expensive in a way that doesn't buy back proportional performance. Infrastructure cost modeling puts a 10 PB all-NVMe deployment at a capital expense well into the millions of dollars, plus ongoing annual costs in power and maintenance.
A tiered alternative, say 500 TB of NVMe for the hot tier, a mid-tier SSD warm layer, and a large object storage pool for cold retention, runs considerably less in capital and annual costs, while delivering equivalent performance for the workloads that actually need speed. That's over a million dollars saved in year one alone, for the same active-workload performance.
The hot tier only needs sizing for active checkpoint bandwidth, meaning throughput needed while training is actually running, not total checkpoint volume across the life of a project. The cold tier handles long-term retention at a fraction of the per-terabyte cost, with no performance penalty, because nobody reads a six-month-old checkpoint at NVMe speed.
Move datasets onto hot storage a day ahead of a scheduled job. Keep only the three most recent checkpoints on the hot tier and archive everything older. Demote checkpoints to cold storage automatically 30 days after a project wraps. None of this is exotic. It's housekeeping, but it's housekeeping with a million-dollar difference attached.
One more cost hides in cloud deployments that on-premises infrastructure sidesteps entirely: egress. Downloading models and checkpoints out of a cloud environment runs $0.05 to $0.12 per gigabyte, and at routine checkpoint sizes of 500 gigabytes or more, that egress bill can rival the cost of the storage capacity itself. Combined with the active-storage multiplier from earlier, teams running several experiments side by side need storage that scales with the number of experiments as well as the size of the model.
GPUDirect Storage and RDMA: how the I/O path itself determines checkpoint throughput
The conventional path for writing a checkpoint routes everything through CPU memory before it touches storage. At checkpoint sizes north of 500 GB, that detour through the CPU becomes the bottleneck, not the storage device and not the network. Teams that spend money upgrading storage hardware without fixing this path are solving the wrong problem.
GPUDirect Storage, or GDS, fixes it by letting the GPU's own DMA engine write straight to NVMe, bypassing CPU memory. This path demonstrably accelerates checkpoint writes at large checkpoint sizes, removing the CPU as the binding constraint. Extend it over the network with a remote block storage protocol and latency remains low even at large cluster scales, provided the transport is RDMA.
GDS over networked storage requires RDMA, meaning InfiniBand or RoCE with NICs that support GPUDirect RDMA. Standard 100GbE Ethernet without RDMA support cannot use this path. Cloud object stores like S3 or GCS don't support GDS directly either, so data headed there has to stage on local NVMe first, a pipeline step that needs its own planning and its own line in the budget.
Between InfiniBand and RoCE, the trade-off is fairly clean, and the right answer depends on what a team is optimizing for. InfiniBand has traditionally won on lowest latency and highest bandwidth, at higher infrastructure cost. RoCE runs over commodity Ethernet with RDMA layered on top, and when configured correctly it scales nearly as well as InfiniBand for a fraction of the price. RoCE has shown up in large training runs, including Meta's Llama models and IBM's Granite models, precisely because it offers near-InfiniBand performance over commodity Ethernet infrastructure. Separate benchmarking from SkyPilot found that choosing the right storage backend alone can nearly double checkpointing speed, independent of which network fabric sits underneath it.
DeepSeek's Fire-Flyer File System, built specifically for LLM training and inference workloads, is a concrete example of a storage system designed around this exact problem, and it shows up in the reference list of recent RDMA communication research. Getting the I/O path right recovers GPU time that a slower path would otherwise burn on every synchronous write, and that recovered time is a return on infrastructure spend that can actually be computed, not just claimed.
Checkpoint storage in the context of total training spend
Scale changes what "negligible" means, and pretending otherwise is how teams get caught flat-footed. For a compact domain-specific model with total training compute around $50,000, checkpoint storage really is a rounding error. A 2 to 10% storage share of total spend comes out to a few thousand dollars, not worth an engineer's attention.
At mid-range scale, a 70B-class model with total compute spend running well into six or seven figures, that same 2 to 10% share becomes a budget line with real weight behind it. Past that point, further into frontier-scale training, the mechanics covered here (checkpoint size, cadence, tiering, the I/O path) stop being background detail. A storage bill can either stay an afterthought or quietly rival the savings a team fought hard to find elsewhere in its compute budget.
Checkpoints are the safety net under runs that cost millions of dollars and take weeks to finish, and they should stay that way. The engineering work is deciding how big, how often, and where those checkpoints live, and getting those three answers right before the bill arrives is what separates a team that planned for this from one explaining, after the fact, why it mattered.
