tokens/sec tuning ideas with TensorRT-LLM

[torchtrack24]

Posting this in case it saves someone else a little time.

I'm not trying to build a perfect architecture diagram. I just want a sane starting point around vLLM and 5090.

I'm especially interested in mistakes worth avoiding.

[llmpeak]

Good point. The pattern around tail latency has been pretty similar for me.

[oxidemixer]

Good point. The pattern around tail latency has been pretty similar for me.

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with DeepSeek and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[thetashed24]

that fixed it for me too.

[quietsupply11]

That matches what I was seeing. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[dockridge]

Maybe, but I had the opposite result. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.

[quantyard]

I'd start with the simpler stack.

[macropoint]

I'd approach it in a boring order. If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious.

[quiettoken]

Need a little more detail here. Are you testing this on 4090 or something else?

[tensorport]

Thanks, that helps. That matches what I saw on GB200.