router in front of vLLM: anyone testing this?

[southpatch]

Maybe, but I had the opposite result. I've seen people blame batch size when the real issue was much more mundane.

I think the bottleneck is somewhere else. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Could be a workload difference, but I wouldn't generalize too fast from one setup.