+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Optimization & Performance (https://www.datasculpting.ai/inference/forumdisplay.php?fid=10)
+--- Forum: Context / KV Cache (https://www.datasculpting.ai/inference/forumdisplay.php?fid=41)
+--- Thread: prefix caching on repeated workflows with TensorRT-LLM (/showthread.php?tid=1442)
Pages:
1
2
Re: prefix caching on repeated workflows with TensorRT-LLM - cacheline - 10-18-2026
Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.
Re: prefix caching on repeated workflows with TensorRT-LLM - cacheline - 10-19-2026
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefix caching. Once you know the plain path works, then add the faster path back in piece by piece.
Re: prefix caching on repeated workflows with TensorRT-LLM - turbogrid11 - 10-19-2026
Quote:A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefix caching. Once you k
Not sure I agree there. I get why that seems right, but on Threadripper box the bottleneck felt elsewhere.
Re: prefix caching on repeated workflows with TensorRT-LLM - cacheline - 10-19-2026
Quote:[quote]A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefix caching. Onc
A couple things I'd check first. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.
Re: prefix caching on repeated workflows with TensorRT-LLM - kernelghost - 10-19-2026
Need a little more detail here. What model, what runtime, and roughly what context size?
Re: prefix caching on repeated workflows with TensorRT-LLM - cacheline - 10-19-2026
That matches what I was seeing. That matches what I saw on Threadripper box. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: prefix caching on repeated workflows with TensorRT-LLM - kernelghost - 10-19-2026
A couple questions before I guess too hard. Is this local-only or are you routing anything externally?
Re: prefix caching on repeated workflows with TensorRT-LLM - sparrowhall - 10-19-2026
I hit almost the same issue last night.
prefix caching on repeated workflows with TensorRT-LLM - cacheline - 10-19-2026
This keeps coming up in conversation so I figured I'd start a thread.
The more I work with local setups, the more it seems like prefix caching decides everything downstream.
Maybe obvious to some of you, but it clicked for me this week.
Re: prefix caching on repeated workflows with TensorRT-LLM - cacheline - 10-19-2026
What exact setup are you running? Is this local-only or are you routing anything externally?