KV cache compression worth it for Mistral

[dockbrook21]

same here on my side.

Interesting, I hadn't tried that. The pattern around batch size has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.