KV cache compression worth it

[quantyard]

Tried to reproduce it just now. On mini PC with PyTorch, I saw basically the same behavior. The main thing that moved was latency, not quality.

[mlxgarden]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Phi and compare from there.