router in front of vLLM: early impressions

[batchsize]

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about MoE serving.

I'm especially interested in mistakes worth avoiding.

This is mostly for local work rather than public serving.

[phasestream53]

Interesting, I hadn't tried that. That matches what I saw on M3 Max. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[cacheline]

Interesting, I hadn't tried that. The pattern around prefill/decode split has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[oxidemixer]

worth checking the driver first.

[thetashed24]

Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[cacheline]

subscribed, curious where this lands.

Need a little more detail here. Is this local-only or are you routing anything externally?

[batchsize]

I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher. I expected the model choice to matter more than it did.

[cudacrate]

I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher. I expected the model choice to matter more than it did.

Thanks, that helps. The pattern around workstation thermals has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[llmpeak]

worth checking the driver first.

[vectorport]

That matches what I was seeing. That matches what I saw on GB200. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.