+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: Other Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=39)
+--- Thread: KV cache compression worth it (/showthread.php?tid=400)
Pages:
1
2
KV cache compression worth it - cachelab - 09-22-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
Just sharing a setup that ended up working reasonably well: Gemma, TensorRT-LLM, 5090, with a lot of attention on prefill/decode split.
Feel free to tear it apart if you see something dumb.
Re: KV cache compression worth it - mangovault67 - 09-22-2026
subscribed, curious where this lands.
Re: KV cache compression worth it - circuitwire58 - 09-22-2026
Interesting, I hadn't tried that. That matches what I saw on M3 Max. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: KV cache compression worth it - turboscope5 - 09-22-2026
I hit almost the same issue last night.
Re: KV cache compression worth it - fastquill74 - 09-22-2026
Good point. The pattern around reranking has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing. I keep promising myself this is the last rebuild.
Re: KV cache compression worth it - quantshed - 09-22-2026
Maybe, but I had the opposite result. I get why that seems right, but on 7900 XTX the bottleneck felt elsewhere.
Re: KV cache compression worth it - circuitwire58 - 09-23-2026
Quote:Maybe, but I had the opposite result. I get why that seems right, but on 7900 XTX the bottleneck felt elsewhere.
Can you fill in a bit more context? Is this local-only or are you routing anything externally?
Re: KV cache compression worth it - audiorunner74 - 09-23-2026
I'd approach it in a boring order. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
I'm happy to test one more variant if you want a second data point.
Re: KV cache compression worth it - edgequill - 09-23-2026
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at batch size. Once you know the plain path works, then add the faster path back in piece by piece.
Re: KV cache compression worth it - llm_wren - 09-24-2026
same here on my side.