routing between local and hosted with vLLM

[rasterpoint]

Good point. That matches what I saw on dual 3090. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread lo

I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.