prefix caching in llama.cpp with PyTorch

[visionyard23]

I think the bottleneck is somewhere else. I've seen people blame KV cache pressure when the real issue was much more mundane. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[phasescope]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at reranking.

[bytesupply76]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Kimi and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.