prefix caching in llama.cpp

[vrambandit]

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about tail latency.

Would rather learn from your scars than collect my own all at once.

This is mostly for local work rather than public serving.

[vrambandit]

Thanks, that helps. The pattern around prefix caching has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer. Turns out 'just one more test' was a lie.

[pixelcrate82]

Maybe, but I had the opposite result. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the simpler explanation before adopting the more interesting one.

[phasescope]

Maybe, but I had the opposite result. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Wort

Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal.

[vrambandit]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at quant drift. Once you know the plain path works, then add the faster path back in piece by piece.

[embershed99]

I'd start with the simpler stack.

[visionyard23]

I hit almost the same issue last night.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[edgequill]

Need a little more detail here. Is this local-only or are you routing anything externally?

[embershed99]

following this because I'm in the same boat.

[northcraft90]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at small local stacks. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.