KV cache compression worth it: early impressions

[sparrowhall]

Spent some time on this today and wanted to compare notes.

I've gone back and forth between Ollama and LM Studio for Qwen. The interesting difference for me has been token cost.

Interested in tradeoffs more than fan loyalty. Turns out 'just one more test' was a lie.

[mangowire]

What worked for me was pretty simple. If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious.