+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: LLM Models (https://www.datasculpting.ai/inference/forumdisplay.php?fid=12)
+--- Forum: Open Weight (https://www.datasculpting.ai/inference/forumdisplay.php?fid=49)
+---- Forum: Recipes & Configurations (https://www.datasculpting.ai/inference/forumdisplay.php?fid=52)
+---- Thread: simple router policy ideas: worth it? (/showthread.php?tid=619)
simple router policy ideas: worth it? - modrunner27 - 09-30-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
Just sharing a setup that ended up working reasonably well: Qwen, TensorRT-LLM, Threadripper box, with a lot of attention on tail latency.
Feel free to tear it apart if you see something dumb.
I care more about stability than squeezing the last possible tokens/sec out of it.
I expected the biggest difference to come from the model, but the stack around it mattered at least as much.
Re: simple router policy ideas: worth it? - sanescope - 09-30-2026
My first guess would be this: If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.
Re: simple router policy ideas: worth it? - deepseeked - 09-30-2026
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at continuous batching.
Re: simple router policy ideas: worth it? - batchyard - 10-01-2026
I think the bottleneck is somewhere else. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Worth testing the simpler explanation before adopting the more interesting one.
Re: simple router policy ideas: worth it? - mangovault67 - 10-01-2026
Can you fill in a bit more context? Is this local-only or are you routing anything externally?
Re: simple router policy ideas: worth it? - amberrunner - 10-01-2026
Quote:Can you fill in a bit more context? Is this local-only or are you routing anything externally?
I hit almost the same issue last night.
Re: simple router policy ideas: worth it? - minthall - 10-01-2026
worth checking the driver first.
Re: simple router policy ideas: worth it? - tensorkite - 10-02-2026
Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly.
Re: simple router policy ideas: worth it? - macropoint - 10-03-2026
Quote:Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly.
Ran a quick check on my box. The issue went away once I changed one setting and restarted the stack cleanly.