+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Optimization & Performance (https://www.datasculpting.ai/inference/forumdisplay.php?fid=10)
+--- Forum: Context / KV Cache (https://www.datasculpting.ai/inference/forumdisplay.php?fid=41)
+--- Thread: prefix caching on repeated workflows with TensorRT-LLM (/showthread.php?tid=1442)
Pages:
1
2
Re: prefix caching on repeated workflows with TensorRT-LLM - amberplane - 10-19-2026
What exact setup are you running? Are you testing this on mini PC or something else?
Re: prefix caching on repeated workflows with TensorRT-LLM - turboscope - 10-19-2026
Quote:What exact setup are you running? Is this local-only or are you routing anything externally?
I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.