routing between local and hosted with vLLM

[cudaline]

Did a bit of testing and thought I'd post the result here.

Short version: MLX behaved well for me, but only after I stopped ignoring small local stacks.

Nothing scientific here, just practical notes from a real setup.

[cacheworks]

Did a bit of testing and thought I'd post the result here.

A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?

[phasespark]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Phi and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.

[latentpeak22]

That matches what I was seeing. The pattern around tail latency has been pretty similar for me.

[bytegarden73]

that fixed it for me too.

[macrotinker]

Thanks, that helps. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[fusedhall]

Can you fill in a bit more context? What model, what runtime, and roughly what context size?

[tensorworks]

Good point. That matches what I saw on dual 3090. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[rasterpoint]

Good point. That matches what I saw on dual 3090. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread lo

I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[torchshed]

[quote]Good point. That matches what I saw on dual 3090. Worth rechecking your defaults though, because one bad assumption can make the rest of the th

What exact setup are you running? What model, what runtime, and roughly what context size?