Scaling Economics
Storage Scaling Bottlenecks in Large LLM Inference Clusters
KV cache grows without bounds, forcing engineers to choose which storage tier fails first.
Marcus Dell'Acqua
· 10 min read
The Burn Layer — Page 3
Scaling Economics
Unit Cost Curves as LLM Inference Scales From 10 to 10,000 GPUs
Scaling LLM inference isn't linear: utilization and scheduling matter more than GPU count.
Dae-Jung Kwon10 min read
Systems Design
Data Preprocessing Pipeline Design for LLM Training Throughput
Careful stage-by-stage design determines whether GPUs train efficiently or starve for tokens.
Marcus Dell'Acqua11 min read
Systems Design
High-Throughput Checkpoint Design for LLM Training at Scale
Failures at massive scale demand checkpoints so frequent that I/O overhead dominates training time.
Marcus Dell'Acqua11 min read
Systems Design
Tiered Memory Architecture for Long-Context LLM Inference
A tiered memory hierarchy solves long-context LLM serving bottlenecks.
Celeste Oforiwaa11 min read
Systems Design
Paged Attention and Memory Efficiency in Production LLM Serving
How PagedAttention solved the memory crisis that made LLM serving uneconomical.
Celeste Oforiwaa13 min read
Systems Design
BF16 and FP8 Compression Strategies for KV Cache Storage
Compressing KV cache with FP8 halves memory while preserving accuracy.
Marcus Dell'Acqua9 min read
Total Cost of Inference
Power and Cooling Costs in GPU Inference Data Centers
Power and cooling infrastructure now costs as much as the GPUs themselves.
Dae-Jung Kwon11 min read
Total Cost of Inference
Operational Labor Cost in AI Infrastructure Teams
Staffing and managing AI infrastructure now costs as much as the compute itself.
Priya Subramaniam8 min read
Total Cost of Inference
Networking Cost in Distributed LLM Inference Clusters
KV cache transfers between GPU pools drive network costs more than computation or model size.
Dae-Jung Kwon10 min read
Total Cost of Inference
Checkpoint Storage Cost During LLM Training Runs
Optimizer state, not weights, drives checkpoint costs during LLM training.
Celeste Oforiwaa11 min read
Total Cost of Inference
Hidden Costs of Running Open-Source LLMs in Production
Self-hosted LLMs cost 95% more than their free model weights suggest.
Dae-Jung Kwon11 min read
