router in front of vLLM: anyone testing this?

[batchcraft]

Maybe, but I had the opposite result. I've seen people blame INT4 when the real issue was much more mundane. Could be a workload difference, but I wou

I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the simpler explanation before adopting the more interesting one.