router in front of vLLM

[batchsize]

Did a bit of testing and thought I'd post the result here.

I tried this with Mixtral using SGLang on Threadripper box. The headline result was better than I expected, but HBM constraints still ended up being the limiting factor.

Curious if anyone else landed in the same place.

I care more about stability than squeezing the last possible tokens/sec out of it.

[signalcraft14]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at continuous batching. Not glamorous advice, but repeatable baselines save a lot of time.

[mangofield33]

What worked for me was pretty simple. On A6000, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

If you post your exact settings, people here can probably narrow it down pretty fast.