+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: Other Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=39)
+--- Thread: tokens/sec tuning ideas with TensorRT-LLM (/showthread.php?tid=799)
Pages:
1
2
tokens/sec tuning ideas with TensorRT-LLM - torchtrack24 - 10-05-2026
Posting this in case it saves someone else a little time.
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around vLLM and 5090.
I'm especially interested in mistakes worth avoiding.
Re: tokens/sec tuning ideas with TensorRT-LLM - llmpeak - 10-05-2026
Good point. The pattern around tail latency has been pretty similar for me.
Re: tokens/sec tuning ideas with TensorRT-LLM - oxidemixer - 10-05-2026
Quote:Good point. The pattern around tail latency has been pretty similar for me.
What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with DeepSeek and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: tokens/sec tuning ideas with TensorRT-LLM - thetashed24 - 10-08-2026
that fixed it for me too.
Re: tokens/sec tuning ideas with TensorRT-LLM - quietsupply11 - 10-10-2026
That matches what I was seeing. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: tokens/sec tuning ideas with TensorRT-LLM - dockridge - 10-13-2026
Maybe, but I had the opposite result. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.
Re: tokens/sec tuning ideas with TensorRT-LLM - quantyard - 10-13-2026
I'd start with the simpler stack.
Re: tokens/sec tuning ideas with TensorRT-LLM - macropoint - 10-13-2026
I'd approach it in a boring order. If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious.
Re: tokens/sec tuning ideas with TensorRT-LLM - quiettoken - 10-14-2026
Need a little more detail here. Are you testing this on 4090 or something else?
Re: tokens/sec tuning ideas with TensorRT-LLM - tensorport - 10-14-2026
Thanks, that helps. That matches what I saw on GB200.