+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: llama.cpp / GGUF (https://www.datasculpting.ai/inference/forumdisplay.php?fid=35)
+--- Thread: prefix caching in llama.cpp for Phi (/showthread.php?tid=823)
Pages:
1
2
prefix caching in llama.cpp for Phi - mangoquill - 10-07-2026
This keeps coming up in conversation so I figured I'd start a thread.
The more I work with local setups, the more it seems like tail latency decides everything downstream.
Maybe obvious to some of you, but it clicked for me this week.
For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.
Re: prefix caching in llama.cpp for Phi - mellowwire98 - 10-07-2026
Can you fill in a bit more context? What model, what runtime, and roughly what context size?
Re: prefix caching in llama.cpp for Phi - vrambandit - 10-07-2026
Good point. That matches what I saw on 7900 XTX. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: prefix caching in llama.cpp for Phi - latentcraft - 10-07-2026
Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.
Re: prefix caching in llama.cpp for Phi - dockridge - 10-07-2026
Quote:Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not qualit
My first guess would be this: On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Re: prefix caching in llama.cpp for Phi - latentcraft - 10-07-2026
Quote:[quote]Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not
I hit almost the same issue last night.
I'm happy to test one more variant if you want a second data point.
Re: prefix caching in llama.cpp for Phi - dockworks23 - 10-09-2026
A couple questions before I guess too hard. Are you testing this on GB200 or something else?
Re: prefix caching in llama.cpp for Phi - pixelcrate82 - 10-10-2026
I think the bottleneck is somewhere else. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Could be a workload difference, but I wouldn't generalize too fast from one setup.
Re: prefix caching in llama.cpp for Phi - visionyard23 - 10-11-2026
A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.
Re: prefix caching in llama.cpp for Phi - rasterhall - 10-11-2026
A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?