+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: Other Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=39)
+--- Thread: prefill bottlenecks on longer prompts (/showthread.php?tid=472)
prefill bottlenecks on longer prompts - bytegarden73 - 09-25-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
Maybe useful as a starting point: TensorRT-LLM on B200, keeping prefix caching in mind from the start.
Maybe useful as a starting point for newer folks.
This is mostly for local work rather than public serving.
Re: prefill bottlenecks on longer prompts - vrambandit - 09-25-2026
Quote:Thought I'd post a small recipe because a lot of threads ask similar things.
I hit almost the same issue last night.