vLLM vs SGLang for small teams on Threadripper box

[cacheline]

Maybe I'm missing something obvious.

I'm mostly working with Phi in LM Studio on 4090. The part I'm trying to understand is VRAM headroom.

Not looking for the perfect answer, mostly what worked in real use.

[cacheline]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Phi and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.

[batchcraft]

Can you fill in a bit more context? Are you testing this on 7900 XTX or something else?

[batchshed]

Can you fill in a bit more context? Are you testing this on 7900 XTX or something else?

I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.

[llmarch83]

What exact setup are you running? What model, what runtime, and roughly what context size?

If you post your exact settings, people here can probably narrow it down pretty fast.

[batchsize]

What exact setup are you running? What model, what runtime, and roughly what context size?

Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.

[deltapoint17]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at workstation thermals. Turns out 'just one more test' was a lie.

[signalpeak]

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Qwen and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.

[cacheline]

I hit almost the same issue last night.

[signalcraft14]

Interesting, I hadn't tried that. I ended up in about the same place using LM Studio. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.