vLLM vs SGLang for small teams on dual 3090

[griffinloom66]

A couple questions before I guess too hard. Is this local-only or are you routing anything externally?

[promptpoint]

A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?

[tensorkite]

Interesting, I hadn't tried that. I don't think your conclusion is far off.

[amberrunner]

I hit almost the same issue last night.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[thetashed24]

Thanks, that helps. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[fastquill74]

Interesting, I hadn't tried that. I don't think your conclusion is far off.

Not sure I agree there. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.