Total Cost of Inference
The Storage Procurement Decision That Determines Whether Million-Token Serving Is Profitable
Choose your KV cache storage tier before ordering hardware.
Priya Subramaniam
· 9 min read
The Burn Layer
Total Cost of Inference
Why KV Cache Explosion Is a Unit Economics Crisis Before It Is an Engineering Problem
Long-context serving costs scale with memory, not just tokens.
Dae-Jung Kwon8 min read
Build vs. Buy Decisions
On-Premises GPU Clusters vs. Cloud Spot Instances for LLM Training
Ownership breaks even when sustained utilization covers staff, hardware, and infrastructure costs.
Priya Subramaniam9 min read
Build vs. Buy Decisions
The Infrastructure Cost Variables That Proprietary API Benchmarks Deliberately Ignore
Self-hosted LLMs hide massive costs that API benchmarks conveniently omit.
Celeste Oforiwaa8 min read
Build vs. Buy Decisions
Managed API Providers for Open-Weight 70B Models Compared
KV cache management determines whether 70B inference is cheaper than proprietary APIs.
Priya Subramaniam8 min read
Build vs. Buy Decisions
Managed LLM Inference APIs vs. Self-Hosted Open-Source Models
The self-hosted crossover depends on utilization, not just GPU costs.
Celeste Oforiwaa9 min read
Total Cost of Inference
Storage Throughput Bottlenecks That Blow Up Your Per-Token Cost at 128k Context
Storage I/O speed, not GPU compute, determines whether long-context serving stays cost-efficient.
Priya Subramaniam8 min read
GPU Utilization
Prefill vs. Decode GPU Utilization Imbalance
Solve prefill-decode conflicts by matching hardware resources to each phase's actual constraints.
Marcus Dell'Acqua8 min read
GPU Utilization
GPU Utilization Benchmarking Methodology for LLM Inference
Separate prefill and decode measurements to expose compute versus memory constraints.
Celeste Oforiwaa8 min read
GPU Utilization
Data Locality and GPU Utilization in Multi-Node Training
Multi-node training wastes GPU cycles when data moves slowly through the memory hierarchy.
Dae-Jung Kwon8 min read
GPU Utilization
I/O Bottlenecks That Stall GPU Utilization in LLM Serving
KV cache data movement, not compute, creates the GPU utilization ceiling in LLM serving.
Priya Subramaniam8 min read
GPU Utilization
KV Cache Eviction Policies and GPU Utilization Tradeoffs
Choosing which tokens to keep matters far more than how you organize the eviction structure.
Marcus Dell'Acqua8 min read
