simple router policy ideas: what are you seeing?

[fastbench]

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Kimi, vLLM, 7900 XTX, with a lot of attention on batch size.

Maybe useful as a starting point for newer folks. Turns out 'just one more test' was a lie.

[sparrowworks88]

A couple questions before I guess too hard. Are you testing this on mini PC or something else?

[rocmmixer]

Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.

[routeline23]

Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than i

A couple questions before I guess too hard. Are you testing this on dual 3090 or something else?

[routecraft]

worth checking the driver first.

[vrambandit]

Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.

[vectorhouse71]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at router policy. Once you know the plain path works, then add the faster path back in piece by piece.

[kernelgrid]

Ran a quick check on my box. On Threadripper box with PyTorch, I saw basically the same behavior. The main thing that moved was latency, not quality.

[dockember]

Thanks, that helps. That matches what I saw on M3 Max.

[turboyard94]

Appreciate the detail. That matches what I saw on dual 3090. The biggest change for me was simplifying the stack instead of adding one more optimization layer.