simple router policy ideas on B200

[fusedhall56]

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: MLX on Threadripper box, keeping quant drift in mind from the start.

Feel free to tear it apart if you see something dumb.

I care more about stability than squeezing the last possible tokens/sec out of it.

[vectorport]

Tried to reproduce it just now. On GB200 with MLX, I saw basically the same behavior. The main thing that moved was latency, not quality.

[edgeriver]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at batch size. Not glamorous advice, but repeatable baselines save a lot of time.