+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Optimization & Performance (https://www.datasculpting.ai/inference/forumdisplay.php?fid=10)
+--- Forum: Context / KV Cache (https://www.datasculpting.ai/inference/forumdisplay.php?fid=41)
+--- Thread: prefix caching on repeated workflows on 7900 XTX (/showthread.php?tid=679)
Pages:
1
2
prefix caching on repeated workflows on 7900 XTX - torchdesk - 10-02-2026
Posting this in case it saves someone else a little time.
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around PyTorch and 5090.
Simple and boring is a valid answer.
Re: prefix caching on repeated workflows on 7900 XTX - cacheline - 10-03-2026
I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at token cost. Not glamorous advice, but repeatable baselines save a lot of time.
Re: prefix caching on repeated workflows on 7900 XTX - plaingarden18 - 10-03-2026
My first guess would be this: On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Re: prefix caching on repeated workflows on 7900 XTX - cacheline - 10-04-2026
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at HBM constraints. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it. Turns out 'just one more test' was a lie.
Re: prefix caching on repeated workflows on 7900 XTX - cacheline - 10-04-2026
Appreciate the detail. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: prefix caching on repeated workflows on 7900 XTX - cacheline - 10-04-2026
I think the bottleneck is somewhere else. I get why that seems right, but on 5090 the bottleneck felt elsewhere. Worth testing the simpler explanation before adopting the more interesting one.
Re: prefix caching on repeated workflows on 7900 XTX - byteyard - 10-07-2026
Thanks, that helps. I ended up in about the same place using vLLM.
Re: prefix caching on repeated workflows on 7900 XTX - cacheline - 10-07-2026
My first guess would be this: If you're on Ollama, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.
Re: prefix caching on repeated workflows on 7900 XTX - cacheline - 10-10-2026
Interesting, I hadn't tried that. I ended up in about the same place using MLX. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: prefix caching on repeated workflows on 7900 XTX - cacheline - 10-11-2026
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at consumer GPUs. Once you know the plain path works, then add the faster path back in piece by piece.