KV cache compression worth it

[edgecoast]

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like token cost decides everything downstream.

Maybe obvious to some of you, but it clicked for me this week.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.

[llmpeak]

I don't think that's the whole story. I get why that seems right, but on GB200 the bottleneck felt elsewhere.

[vramscope]

Maybe, but I had the opposite result. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.

[sanevault]

Can you fill in a bit more context? Are you testing this on mini PC or something else?

[macronest]

Can you fill in a bit more context? Are you testing this on mini PC or something else?

Interesting, I hadn't tried that. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[rocmrunner40]

Can you fill in a bit more context? Are you testing this on mini PC or something else?

Need a little more detail here. Do you see the same thing with a smaller batch or shorter prompt?