The Storage Procurement Decision That Determines Whether Million-Token Serving Is Profitable
Choose your KV cache storage tier before ordering hardware.
Priya Subramaniam
Section
Total Cost of Inference
11 stories in Total Cost of Inference.
Why KV Cache Explosion Is a Unit Economics Crisis Before It Is an Engineering Problem
Long-context serving costs scale with memory, not just tokens.
Dae-Jung Kwon
Storage Throughput Bottlenecks That Blow Up Your Per-Token Cost at 128k Context
Storage I/O speed, not GPU compute, determines whether long-context serving stays cost-efficient.
Priya Subramaniam
Power and Cooling Costs in GPU Inference Data Centers
Power and cooling infrastructure now costs as much as the GPUs themselves.
Dae-Jung Kwon
Operational Labor Cost in AI Infrastructure Teams
Staffing and managing AI infrastructure now costs as much as the compute itself.
Priya Subramaniam
Networking Cost in Distributed LLM Inference Clusters
KV cache transfers between GPU pools drive network costs more than computation or model size.
Dae-Jung Kwon
Checkpoint Storage Cost During LLM Training Runs
Optimizer state, not weights, drives checkpoint costs during LLM training.
Celeste Oforiwaa
Hidden Costs of Running Open-Source LLMs in Production
Self-hosted LLMs cost 95% more than their free model weights suggest.
Dae-Jung Kwon
Token Cost Accounting Across Inference Stack Layers
Breaking down token costs reveals four stack layers where savings hide beyond the sticker price.
Priya Subramaniam
NVMe Storage Costs vs. GPU Memory for Inference Offloading
Wafer shortages are making NVMe storage nearly as expensive as GPU memory for inference.
Priya Subramaniam
KV Cache Storage Cost in Long-Context Serving
KV cache size follows a formula that infrastructure teams can measure and control.
Priya Subramaniam