router in front of vLLM

[thetaplane]

Posting this in case it saves someone else a little time.

I'm not trying to build a perfect architecture diagram. I just want a sane starting point around MLX and GB200.

Would rather learn from your scars than collect my own all at once.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

[kernelghost]

Good point. The pattern around reranking has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[routearch]

Thanks, that helps. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[turbosupply83]

Need a little more detail here. Do you see the same thing with a smaller batch or shorter prompt?

[cacheline]

Need a little more detail here. Do you see the same thing with a smaller batch or shorter prompt?

I hit almost the same issue last night.

[ggufvault]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at structured output. Not glamorous advice, but repeatable baselines save a lot of time.

[southspark]

Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal.

[audioforge]

Thanks, that helps. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of t

I don't think that's the whole story. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the simpler explanation before adopting the more interesting one.

[cacheline]

[quote]Thanks, that helps. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the re

What worked for me was pretty simple. If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.

[thetashed24]

[quote][quote]Thanks, that helps. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make

Interesting, I hadn't tried that. That matches what I saw on A6000. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.