+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: LLM Models (https://www.datasculpting.ai/inference/forumdisplay.php?fid=12)
+--- Forum: Open Weight (https://www.datasculpting.ai/inference/forumdisplay.php?fid=49)
+---- Forum: Recipes & Configurations (https://www.datasculpting.ai/inference/forumdisplay.php?fid=52)
+---- Thread: simple router policy ideas: what are you seeing? (/showthread.php?tid=270)
Pages:
1
2
simple router policy ideas: what are you seeing? - fastbench - 09-13-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
Just sharing a setup that ended up working reasonably well: Kimi, vLLM, 7900 XTX, with a lot of attention on batch size.
Maybe useful as a starting point for newer folks. Turns out 'just one more test' was a lie.
Re: simple router policy ideas: what are you seeing? - sparrowworks88 - 09-13-2026
A couple questions before I guess too hard. Are you testing this on mini PC or something else?
Re: simple router policy ideas: what are you seeing? - rocmmixer - 09-15-2026
Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.
Re: simple router policy ideas: what are you seeing? - routeline23 - 09-15-2026
Quote:Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than i
A couple questions before I guess too hard. Are you testing this on dual 3090 or something else?
Re: simple router policy ideas: what are you seeing? - routecraft - 09-15-2026
worth checking the driver first.
Re: simple router policy ideas: what are you seeing? - vrambandit - 09-16-2026
Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.
Re: simple router policy ideas: what are you seeing? - vectorhouse71 - 09-17-2026
I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at router policy. Once you know the plain path works, then add the faster path back in piece by piece.
Re: simple router policy ideas: what are you seeing? - kernelgrid - 09-17-2026
Ran a quick check on my box. On Threadripper box with PyTorch, I saw basically the same behavior. The main thing that moved was latency, not quality.
Re: simple router policy ideas: what are you seeing? - dockember - 09-17-2026
Thanks, that helps. That matches what I saw on M3 Max.
Re: simple router policy ideas: what are you seeing? - turboyard94 - 09-17-2026
Appreciate the detail. That matches what I saw on dual 3090. The biggest change for me was simplifying the stack instead of adding one more optimization layer.