KV Cache Eviction Policies and GPU Utilization Tradeoffs
Choosing which tokens to keep matters far more than how you organize the eviction structure.
Marcus Dell'Acqua
Section
1 story in GPU Utilization.
Choosing which tokens to keep matters far more than how you organize the eviction structure.