prefix caching in llama.cpp for Kimi

[macroforge]

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: LM Studio on A6000, keeping consumer GPUs in mind from the start.

Feel free to tear it apart if you see something dumb.

[vrampilot]

A couple things I'd check first. On GB200, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.

[mangotinker73]

subscribed, curious where this lands.

I'm happy to test one more variant if you want a second data point.

[visionyard23]

subscribed, curious where this lands.

following this because I'm in the same boat.

[embershed99]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Llama and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[latentcraft]

I'd approach it in a boring order. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[macropilot]

I don't think that's the whole story. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.