+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: Other Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=39)
+--- Thread: KV cache compression worth it (/showthread.php?tid=164)
Pages:
1
2
Re: KV cache compression worth it - quantyard - 09-11-2026
Tried to reproduce it just now. On mini PC with PyTorch, I saw basically the same behavior. The main thing that moved was latency, not quality.
Re: KV cache compression worth it - mlxgarden - 09-11-2026
I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Phi and compare from there.