prefix caching on repeated workflows with TensorRT-LLM

[cacheline]

I figured I'd ask here before changing everything.

I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling prefix caching.

If you've tested something similar, what did you keep and what did you stop bothering with?

This is mostly for local work rather than public serving.

[cacheline]

I figured I'd ask here before changing everything.

Good point. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[cacheline]

I figured I'd ask here before changing everything.

A couple things I'd check first. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[quiethouse]

Appreciate the detail. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[cacheline]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at MoE serving. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[thetafield]

Interesting, I hadn't tried that. The pattern around VRAM headroom has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[cacheline]

Interesting, I hadn't tried that. The pattern around VRAM headroom has been pretty similar for me. The biggest change for me was simplifying the stack

That matches what I was seeing. The pattern around small local stacks has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[homelabhenry]

Need a little more detail here. Are you testing this on 7900 XTX or something else?

[emberquill]

My first guess would be this: On GB200, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[mangohouse]

Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.