prefix caching in llama.cpp

[vrambandit]

A couple questions before I guess too hard. What model, what runtime, and roughly what context size?

[mellowwire98]

Maybe, but I had the opposite result. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.

[latentcraft]

A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.