Networking Cost in Distributed LLM Inference Clusters
KV cache transfers between GPU pools drive network costs more than computation or model size.
Vera Osei
Staff Writer
Vera Osei
Vera Osei is a staff writer at The Burn Layer covering total cost of inference. Based in Cape Town, Vera has written for The Burn Layer since 2021.
1 story · Cape Town