+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: LLM Models (https://www.datasculpting.ai/inference/forumdisplay.php?fid=12)
+--- Forum: Open Weight (https://www.datasculpting.ai/inference/forumdisplay.php?fid=49)
+---- Forum: Recipes & Configurations (https://www.datasculpting.ai/inference/forumdisplay.php?fid=52)
+---- Thread: simple router policy ideas (/showthread.php?tid=331)
Pages:
1
2
Re: simple router policy ideas - phasetrack - 09-30-2026
What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at INT4. Not glamorous advice, but repeatable baselines save a lot of time.
Re: simple router policy ideas - griffinloom66 - 09-30-2026
A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Mixtral and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.
Re: simple router policy ideas - quantsupply - 09-30-2026
I don't think that's the whole story. I get why that seems right, but on 4090 the bottleneck felt elsewhere.
Re: simple router policy ideas - paperport64 - 10-03-2026
Quote:I don't think that's the whole story. I get why that seems right, but on 4090 the bottleneck felt elsewhere.
A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?