+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: llama.cpp / GGUF (https://www.datasculpting.ai/inference/forumdisplay.php?fid=35)
+--- Thread: prefix caching in llama.cpp with PyTorch (/showthread.php?tid=839)
Pages:
1
2
Re: prefix caching in llama.cpp with PyTorch - visionyard23 - 10-13-2026
I think the bottleneck is somewhere else. I've seen people blame KV cache pressure when the real issue was much more mundane. Could be a workload difference, but I wouldn't generalize too fast from one setup.
Re: prefix caching in llama.cpp with PyTorch - phasescope - 10-14-2026
What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at reranking.
Re: prefix caching in llama.cpp with PyTorch - bytesupply76 - 10-14-2026
My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Kimi and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.