vLLM vs SGLang for small teams on 5090

[oxidemixer]

This keeps coming up in conversation so I figured I'd start a thread.

One thing I keep noticing is that people underestimate workstation thermals until they actually try to run this stuff for more than one afternoon.

Maybe obvious to some of you, but it clicked for me this week.

[thetashed24]

Thanks, that helps. That matches what I saw on 4090. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[batchsize]

Thanks, that helps. That matches what I saw on 4090. The biggest change for me was simplifying the stack instead of adding one more optimization layer

same here on my side.

[amberrunner]

Good point. The pattern around router policy has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[rocmrunner40]

A couple questions before I guess too hard. Are you testing this on mini PC or something else?

[oxidemixer]

Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.

[batchcraft]

Appreciate the detail. The pattern around batch size has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.