router in front of vLLM: early impressions

[cudacrate]

I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher. I expected the model choice to matter more than it did.

Thanks, that helps. The pattern around workstation thermals has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.