simple router policy ideas: what are you seeing?

[fastquill74]

I think the bottleneck is somewhere else. I've seen people blame MoE serving when the real issue was much more mundane. Worth testing the simpler explanation before adopting the more interesting one.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[modrunner27]

I think the bottleneck is somewhere else. I've seen people blame MoE serving when the real issue was much more mundane. Worth testing the simpler expl

Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?

[audionode8]

That matches what I was seeing. That matches what I saw on B200. The logs were somehow calmer than I was.

[visionforge]

Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.