+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: TensorRT-LLM (https://www.datasculpting.ai/inference/forumdisplay.php?fid=37)
+--- Thread: TensorRT-LLM worth the setup time (/showthread.php?tid=62)
Pages:
1
2
TensorRT-LLM worth the setup time - kernelmike - 08-29-2026
I figured I'd ask here before changing everything.
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling reranking.
Would appreciate practical advice rather than theory.
For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.
Re: TensorRT-LLM worth the setup time - routecrate - 08-29-2026
I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tokens/sec.
Re: TensorRT-LLM worth the setup time - kernelmike - 08-29-2026
Quote:I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tokens/sec.
that fixed it for me too.
Re: TensorRT-LLM worth the setup time - routecrate - 08-29-2026
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at reranking.
Re: TensorRT-LLM worth the setup time - kernelmike - 08-30-2026
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefill/decode split. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: TensorRT-LLM worth the setup time - audiodesk - 08-31-2026
I'd approach it in a boring order. On M3 Max, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: TensorRT-LLM worth the setup time - thetapath - 09-02-2026
What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at embeddings.
Re: TensorRT-LLM worth the setup time - vectorport - 09-02-2026
I hit almost the same issue last night.
Re: TensorRT-LLM worth the setup time - torchsmith25 - 09-05-2026
Interesting, I hadn't tried that. I don't think your conclusion is far off.
Re: TensorRT-LLM worth the setup time - tensorfield88 - 09-06-2026
That matches what I was seeing. The pattern around structured output has been pretty similar for me.