Unit Cost Curves as LLM Inference Scales From 10 to 10,000 GPUs
Scaling LLM inference isn't linear: utilization and scheduling matter more than GPU count.
Vera Lindqvist
Section
1 story in Scaling Economics.
Scaling LLM inference isn't linear: utilization and scheduling matter more than GPU count.