+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: llama.cpp / GGUF (https://www.datasculpting.ai/inference/forumdisplay.php?fid=35)
+--- Thread: prefix caching in llama.cpp (/showthread.php?tid=169)
prefix caching in llama.cpp - bytesupply76 - 09-07-2026
Spent some time on this today and wanted to compare notes.
Two paths both seem viable, but they break down differently once VRAM headroom becomes important.
Trying to avoid rebuilding the whole thing twice.
Re: prefix caching in llama.cpp - vrambandit - 09-07-2026
worth checking the driver first.
Re: prefix caching in llama.cpp - embershed99 - 09-10-2026
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at long context. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it. The logs were somehow calmer than I was.
Re: prefix caching in llama.cpp - thetagrid15 - 09-11-2026
Quote:My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at long context. A lot of these
I'd approach it in a boring order. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: prefix caching in llama.cpp - vrambandit - 09-11-2026
I don't think that's the whole story. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Could be a workload difference, but I wouldn't generalize too fast from one setup.
Re: prefix caching in llama.cpp - phasescope - 09-11-2026
worth checking the driver first.
Re: prefix caching in llama.cpp - pixelcrate82 - 09-11-2026
Good point. I ended up in about the same place using PyTorch. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: prefix caching in llama.cpp - visionyard23 - 09-12-2026
same here on my side.