when TensorRT-LLM clearly wins with SGLang

[routecrate]

Spent some time on this today and wanted to compare notes.

Two paths both seem viable, but they break down differently once prefill/decode split becomes important.

What would you pick if you had to live with it for a few months?

[modbench]

That matches what I was seeing. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[syshouse]

That matches what I was seeing. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one mo

A couple things I'd check first. If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious. I keep promising myself this is the last rebuild.

[tinycontext]

[quote]That matches what I was seeing. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding

I hit almost the same issue last night.

[plainline]

[quote][quote]That matches what I was seeing. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of

same here on my side.

[kernelmike]

I hit almost the same issue last night.

[sysstation69]

I tested this tonight. On GB200 with TensorRT-LLM, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.

If you post your exact settings, people here can probably narrow it down pretty fast.

[rasterbench]

I'd start with the simpler stack.

[vectorhouse71]

I'd start with the simpler stack.

Can you fill in a bit more context? What model, what runtime, and roughly what context size?

[batchcraft]

following this because I'm in the same boat.