+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: Other Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=39)
+--- Thread: KV cache compression worth it (/showthread.php?tid=164)
Pages:
1
2
KV cache compression worth it - batchscope72 - 09-06-2026
Did a bit of testing and thought I'd post the result here.
I tried this with Gemma using PyTorch on Threadripper box. The headline result was better than I expected, but KV cache pressure still ended up being the limiting factor.
Nothing scientific here, just practical notes from a real setup.
Re: KV cache compression worth it - batchshed - 09-06-2026
That matches what I was seeing. That matches what I saw on 5090.
Re: KV cache compression worth it - thetaplane - 09-06-2026
Quote:That matches what I was seeing. That matches what I saw on 5090.
Need a little more detail here. Are you testing this on M3 Max or something else?
Re: KV cache compression worth it - kernelpeak - 09-06-2026
That matches what I was seeing. I don't think your conclusion is far off.
Re: KV cache compression worth it - thetaember38 - 09-06-2026
What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at token cost.
Re: KV cache compression worth it - phaseshed - 09-06-2026
What worked for me was pretty simple. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious.
Re: KV cache compression worth it - bytelab - 09-09-2026
that fixed it for me too. The logs were somehow calmer than I was.
Re: KV cache compression worth it - kernelgrid - 09-11-2026
Interesting, I hadn't tried that. I ended up in about the same place using Ollama. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: KV cache compression worth it - mangopath4 - 09-11-2026
A couple things I'd check first. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.
Re: KV cache compression worth it - batchshed - 09-11-2026
I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefix caching.