Inference
latency vs throughput for small deployments with TensorRT-LLM - Printable Version

+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Optimization & Performance (https://www.datasculpting.ai/inference/forumdisplay.php?fid=10)
+--- Forum: Performance Tuning (https://www.datasculpting.ai/inference/forumdisplay.php?fid=42)
+--- Thread: latency vs throughput for small deployments with TensorRT-LLM (/showthread.php?tid=888)

latency vs throughput for small deployments with TensorRT-LLM - dockridge - 10-07-2026

Maybe I'm missing something obvious.

I'm mostly working with Gemma in SGLang on M3 Max. The part I'm trying to understand is prefill/decode split.

If you've tested something similar, what did you keep and what did you stop bothering with?

Re: latency vs throughput for small deployments with TensorRT-LLM - kernelmike - 10-07-2026

Quote:Maybe I'm missing something obvious.

I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. I expected the model choice to matter more than it did.

Re: latency vs throughput for small deployments with TensorRT-LLM - plainline - 10-10-2026

Thanks, that helps. I ended up in about the same place using vLLM. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

Re: latency vs throughput for small deployments with TensorRT-LLM - stackarch - 10-13-2026

Maybe, but I had the opposite result. I get why that seems right, but on B200 the bottleneck felt elsewhere. Worth testing the simpler explanation before adopting the more interesting one. I keep promising myself this is the last rebuild.

Re: latency vs throughput for small deployments with TensorRT-LLM - bytestack - 10-14-2026

Interesting, I hadn't tried that. The pattern around VRAM headroom has been pretty similar for me.

Re: latency vs throughput for small deployments with TensorRT-LLM - rocmpath - 10-15-2026

What exact setup are you running? Do you see the same thing with a smaller batch or shorter prompt?

Re: latency vs throughput for small deployments with TensorRT-LLM - localpatch78 - 10-18-2026

Good point. That matches what I saw on Threadripper box. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

If you post your exact settings, people here can probably narrow it down pretty fast.

Re: latency vs throughput for small deployments with TensorRT-LLM - vectorpoint - 10-18-2026

Thanks, that helps. I don't think your conclusion is far off.

Re: latency vs throughput for small deployments with TensorRT-LLM - rocmpath - 10-18-2026

That matches what I was seeing. The pattern around quant drift has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

Re: latency vs throughput for small deployments with TensorRT-LLM - turboscope5 - 10-18-2026

Maybe, but I had the opposite result. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Worth testing the simpler explanation before adopting the more interesting one.