KV cache compression worth it: worth it?

[marbletrack]

Thanks, that helps. The pattern around consumer GPUs has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.