+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: vLLM / SGLang (https://www.datasculpting.ai/inference/forumdisplay.php?fid=36)
+--- Thread: router in front of vLLM (/showthread.php?tid=1255)
Pages:
1
2
router in front of vLLM - thetaplane - 10-16-2026
Posting this in case it saves someone else a little time.
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around MLX and GB200.
Would rather learn from your scars than collect my own all at once.
My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.
Re: router in front of vLLM - kernelghost - 10-16-2026
Good point. The pattern around reranking has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: router in front of vLLM - routearch - 10-18-2026
Thanks, that helps. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: router in front of vLLM - turbosupply83 - 10-18-2026
Need a little more detail here. Do you see the same thing with a smaller batch or shorter prompt?
Re: router in front of vLLM - cacheline - 10-18-2026
Quote:Need a little more detail here. Do you see the same thing with a smaller batch or shorter prompt?
I hit almost the same issue last night.
Re: router in front of vLLM - ggufvault - 10-19-2026
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at structured output. Not glamorous advice, but repeatable baselines save a lot of time.
Re: router in front of vLLM - southspark - 10-19-2026
Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal.
Re: router in front of vLLM - audioforge - 10-19-2026
Quote:Thanks, that helps. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of t
I don't think that's the whole story. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the simpler explanation before adopting the more interesting one.
Re: router in front of vLLM - cacheline - 10-19-2026
Quote:[quote]Thanks, that helps. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the re
What worked for me was pretty simple. If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.
Re: router in front of vLLM - thetashed24 - 10-19-2026
Quote:[quote][quote]Thanks, that helps. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make
Interesting, I hadn't tried that. That matches what I saw on A6000. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.