+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: llama.cpp / GGUF (https://www.datasculpting.ai/inference/forumdisplay.php?fid=35)
+--- Thread: prefix caching in llama.cpp (/showthread.php?tid=1391)
Pages:
1
2
prefix caching in llama.cpp - vectorport - 10-17-2026
Spent some time on this today and wanted to compare notes.
I've gone back and forth between vLLM and TensorRT-LLM for FLUX. The interesting difference for me has been long context.
Trying to avoid rebuilding the whole thing twice.
Re: prefix caching in llama.cpp - embershed99 - 10-18-2026
Quote:Spent some time on this today and wanted to compare notes.
A couple things I'd check first. If you're on Ollama, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it. Turns out 'just one more test' was a lie.
Re: prefix caching in llama.cpp - mangotinker73 - 10-18-2026
worth checking the driver first.
Re: prefix caching in llama.cpp - vrambandit - 10-18-2026
Interesting, I hadn't tried that. That matches what I saw on GB200. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: prefix caching in llama.cpp - kernelrunner - 10-18-2026
Quote:A couple questions before I guess too hard. What model, what runtime, and roughly what context size?
I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.
Re: prefix caching in llama.cpp - embershed99 - 10-18-2026
Quote:Maybe, but I had the opposite result. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.
A couple things I'd check first. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: prefix caching in llama.cpp - thetagrid15 - 10-18-2026
Ran a quick check on my box. On dual 3090 with MLX, I saw basically the same behavior. The main thing that moved was latency, not quality.
Re: prefix caching in llama.cpp - northgrid33 - 10-19-2026
Appreciate the detail. That matches what I saw on A6000.
Re: prefix caching in llama.cpp - bytesupply76 - 10-19-2026
Ran a quick check on my box. On mini PC with llama.cpp, I saw basically the same behavior. I expected the model choice to matter more than it did.
Re: prefix caching in llama.cpp - northcraft90 - 10-19-2026
Interesting, I hadn't tried that. I ended up in about the same place using llama.cpp. The biggest change for me was simplifying the stack instead of adding one more optimization layer.