latency vs throughput for small deployments with TensorRT-LLM

[dockridge]

Maybe I'm missing something obvious.

I'm mostly working with Gemma in SGLang on M3 Max. The part I'm trying to understand is prefill/decode split.

If you've tested something similar, what did you keep and what did you stop bothering with?

[kernelmike]

Maybe I'm missing something obvious.

I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. I expected the model choice to matter more than it did.

[plainline]

Thanks, that helps. I ended up in about the same place using vLLM. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[stackarch]

Maybe, but I had the opposite result. I get why that seems right, but on B200 the bottleneck felt elsewhere. Worth testing the simpler explanation before adopting the more interesting one. I keep promising myself this is the last rebuild.

[bytestack]

Interesting, I hadn't tried that. The pattern around VRAM headroom has been pretty similar for me.

[rocmpath]

What exact setup are you running? Do you see the same thing with a smaller batch or shorter prompt?

[localpatch78]

Good point. That matches what I saw on Threadripper box. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

If you post your exact settings, people here can probably narrow it down pretty fast.

[vectorpoint]

Thanks, that helps. I don't think your conclusion is far off.

[rocmpath]

That matches what I was seeing. The pattern around quant drift has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[turboscope5]

Maybe, but I had the opposite result. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Worth testing the simpler explanation before adopting the more interesting one.