+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: Other Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=39)
+--- Thread: KV cache compression worth it: worth it? (/showthread.php?tid=13)
Pages:
1
2
KV cache compression worth it: worth it? - mintquill - 08-23-2026
Has anyone else run into this?
I'm mostly working with Mistral in LM Studio on dual 3090. The part I'm trying to understand is VRAM headroom.
Would appreciate practical advice rather than theory.
Re: KV cache compression worth it: worth it? - macropilot - 08-23-2026
Appreciate the detail. I ended up in about the same place using Ollama.
Re: KV cache compression worth it: worth it? - audionode8 - 08-23-2026
Maybe, but I had the opposite result. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.
Re: KV cache compression worth it: worth it? - quiettoken - 08-23-2026
What worked for me was pretty simple. If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.
Re: KV cache compression worth it: worth it? - vrampilot - 08-24-2026
What exact setup are you running? Is this local-only or are you routing anything externally? Turns out 'just one more test' was a lie.
Re: KV cache compression worth it: worth it? - phasescope - 08-25-2026
same here on my side.
Re: KV cache compression worth it: worth it? - quiethouse - 08-26-2026
Need a little more detail here. What model, what runtime, and roughly what context size?
Re: KV cache compression worth it: worth it? - sparrowworks88 - 08-26-2026
That matches what I was seeing. I ended up in about the same place using LM Studio.
Re: KV cache compression worth it: worth it? - phasenode - 08-27-2026
Good point. The pattern around tokens/sec has been pretty similar for me.
Re: KV cache compression worth it: worth it? - marbleyard32 - 08-27-2026
Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the simpler explanation before adopting the more interesting one.
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.