routing between local and hosted with vLLM

[torchshed]

[quote]Good point. That matches what I saw on dual 3090. Worth rechecking your defaults though, because one bad assumption can make the rest of the th

What exact setup are you running? What model, what runtime, and roughly what context size?