+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: vLLM / SGLang (https://www.datasculpting.ai/inference/forumdisplay.php?fid=36)
+--- Thread: router in front of vLLM: early impressions (/showthread.php?tid=1368)
Pages:
1
2
router in front of vLLM: early impressions - batchsize - 10-16-2026
Posting this in case it saves someone else a little time.
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about MoE serving.
I'm especially interested in mistakes worth avoiding.
This is mostly for local work rather than public serving.
Re: router in front of vLLM: early impressions - phasestream53 - 10-16-2026
Interesting, I hadn't tried that. That matches what I saw on M3 Max. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: router in front of vLLM: early impressions - cacheline - 10-17-2026
Interesting, I hadn't tried that. The pattern around prefill/decode split has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: router in front of vLLM: early impressions - oxidemixer - 10-18-2026
worth checking the driver first.
Re: router in front of vLLM: early impressions - thetashed24 - 10-18-2026
Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.
Re: router in front of vLLM: early impressions - cacheline - 10-18-2026
Quote:subscribed, curious where this lands.
Need a little more detail here. Is this local-only or are you routing anything externally?
Re: router in front of vLLM: early impressions - batchsize - 10-18-2026
I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher. I expected the model choice to matter more than it did.
Re: router in front of vLLM: early impressions - cudacrate - 10-18-2026
Quote:I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher. I expected the model choice to matter more than it did.
Thanks, that helps. The pattern around workstation thermals has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: router in front of vLLM: early impressions - llmpeak - 10-18-2026
worth checking the driver first.
Re: router in front of vLLM: early impressions - vectorport - 10-19-2026
That matches what I was seeing. That matches what I saw on GB200. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.