+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: LLM Models (https://www.datasculpting.ai/inference/forumdisplay.php?fid=12)
+--- Forum: Open Weight (https://www.datasculpting.ai/inference/forumdisplay.php?fid=49)
+---- Forum: Recipes & Configurations (https://www.datasculpting.ai/inference/forumdisplay.php?fid=52)
+---- Thread: simple router policy ideas (/showthread.php?tid=331)
Pages:
1
2
simple router policy ideas - kernelgrid - 09-18-2026
This keeps coming up in conversation so I figured I'd start a thread.
The more I work with local setups, the more it seems like INT4 decides everything downstream.
Curious whether that lines up with what other people are seeing.
This is mostly for local work rather than public serving.
Re: simple router policy ideas - sparrowcraft - 09-18-2026
A couple questions before I guess too hard. Are you testing this on mini PC or something else?
Re: simple router policy ideas - quantghost - 09-18-2026
That matches what I was seeing. That matches what I saw on A6000. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
I'm happy to test one more variant if you want a second data point.
Re: simple router policy ideas - cacheline - 09-21-2026
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at NVFP4. Once you know the plain path works, then add the faster path back in piece by piece.
Re: simple router policy ideas - modgarden - 09-21-2026
What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at router policy. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: simple router policy ideas - torchkite - 09-24-2026
I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Llama and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: simple router policy ideas - oxidestream - 09-27-2026
that fixed it for me too.
Re: simple router policy ideas - fusedhall56 - 09-27-2026
Quote:that fixed it for me too.
What exact setup are you running? Is this local-only or are you routing anything externally?
Re: simple router policy ideas - kernelrunner - 09-27-2026
Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?
If you post your exact settings, people here can probably narrow it down pretty fast.
Re: simple router policy ideas - embershed99 - 09-30-2026
Need a little more detail here. What model, what runtime, and roughly what context size?