+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: LLM Models (https://www.datasculpting.ai/inference/forumdisplay.php?fid=12)
+--- Forum: Open Weight (https://www.datasculpting.ai/inference/forumdisplay.php?fid=49)
+---- Forum: Recipes & Configurations (https://www.datasculpting.ai/inference/forumdisplay.php?fid=52)
+---- Thread: routing between local and hosted with vLLM (/showthread.php?tid=542)
Pages:
1
2
routing between local and hosted with vLLM - cudaline - 09-28-2026
Did a bit of testing and thought I'd post the result here.
Short version: MLX behaved well for me, but only after I stopped ignoring small local stacks.
Nothing scientific here, just practical notes from a real setup.
Re: routing between local and hosted with vLLM - cacheworks - 09-29-2026
Quote:Did a bit of testing and thought I'd post the result here.
A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?
Re: routing between local and hosted with vLLM - phasespark - 09-29-2026
My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Phi and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.
Re: routing between local and hosted with vLLM - latentpeak22 - 09-30-2026
That matches what I was seeing. The pattern around tail latency has been pretty similar for me.
Re: routing between local and hosted with vLLM - bytegarden73 - 09-30-2026
that fixed it for me too.
Re: routing between local and hosted with vLLM - macrotinker - 09-30-2026
Thanks, that helps. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: routing between local and hosted with vLLM - fusedhall - 09-30-2026
Can you fill in a bit more context? What model, what runtime, and roughly what context size?
Re: routing between local and hosted with vLLM - tensorworks - 09-30-2026
Good point. That matches what I saw on dual 3090. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: routing between local and hosted with vLLM - rasterpoint - 09-30-2026
Quote:Good point. That matches what I saw on dual 3090. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread lo
I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.
Re: routing between local and hosted with vLLM - torchshed - 10-01-2026
Quote:[quote]Good point. That matches what I saw on dual 3090. Worth rechecking your defaults though, because one bad assumption can make the rest of the th
What exact setup are you running? What model, what runtime, and roughly what context size?