KV cache compression worth it

[batchscope72]

Did a bit of testing and thought I'd post the result here.

I tried this with Gemma using PyTorch on Threadripper box. The headline result was better than I expected, but KV cache pressure still ended up being the limiting factor.

Nothing scientific here, just practical notes from a real setup.

[batchshed]

That matches what I was seeing. That matches what I saw on 5090.

[thetaplane]

That matches what I was seeing. That matches what I saw on 5090.

Need a little more detail here. Are you testing this on M3 Max or something else?

[kernelpeak]

That matches what I was seeing. I don't think your conclusion is far off.

[thetaember38]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at token cost.

[phaseshed]

What worked for me was pretty simple. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious.

[bytelab]

that fixed it for me too. The logs were somehow calmer than I was.

[kernelgrid]

Interesting, I hadn't tried that. I ended up in about the same place using Ollama. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[mangopath4]

A couple things I'd check first. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

[batchshed]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefix caching.