KV cache compression worth it: worth it?

[mintquill]

Has anyone else run into this?

I'm mostly working with Mistral in LM Studio on dual 3090. The part I'm trying to understand is VRAM headroom.

Would appreciate practical advice rather than theory.

[macropilot]

Appreciate the detail. I ended up in about the same place using Ollama.

[audionode8]

Maybe, but I had the opposite result. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.

[quiettoken]

What worked for me was pretty simple. If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.

[vrampilot]

What exact setup are you running? Is this local-only or are you routing anything externally? Turns out 'just one more test' was a lie.

[phasescope]

same here on my side.

[quiethouse]

Need a little more detail here. What model, what runtime, and roughly what context size?

[sparrowworks88]

That matches what I was seeing. I ended up in about the same place using LM Studio.

[phasenode]

Good point. The pattern around tokens/sec has been pretty similar for me.

[marbleyard32]

Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the simpler explanation before adopting the more interesting one.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.