vLLM vs SGLang for small teams

[cacheline]

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: LM Studio on mini PC, keeping continuous batching in mind from the start.

Feel free to tear it apart if you see something dumb.

[oxidemixer]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with FLUX and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.

[deltapoint17]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with FLUX and compare from there. Not glamorous advice, but

Can you fill in a bit more context? Is this local-only or are you routing anything externally?

[quantghost]

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with FLUX and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

If you post your exact settings, people here can probably narrow it down pretty fast.

[cacheline]

Interesting, I hadn't tried that. The pattern around KV cache pressure has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[turboarch95]

following this because I'm in the same boat.

[audioforge]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at NVFP4. Once you know the plain path works, then add the faster path back in piece by piece.

[turboarch95]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at batch size. Not glamorous advice, but repeatable baselines save a lot of time.

[deltapoint17]

What exact setup are you running? Do you see the same thing with a smaller batch or shorter prompt?

[ggufvault]

following this because I'm in the same boat. The logs were somehow calmer than I was.