simple router policy ideas: worth it?

[modrunner27]

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Qwen, TensorRT-LLM, Threadripper box, with a lot of attention on tail latency.

Feel free to tear it apart if you see something dumb.

I care more about stability than squeezing the last possible tokens/sec out of it.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.

[sanescope]

My first guess would be this: If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.

[deepseeked]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at continuous batching.

[batchyard]

I think the bottleneck is somewhere else. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Worth testing the simpler explanation before adopting the more interesting one.

[mangovault67]

Can you fill in a bit more context? Is this local-only or are you routing anything externally?

[amberrunner]

Can you fill in a bit more context? Is this local-only or are you routing anything externally?

I hit almost the same issue last night.

[minthall]

worth checking the driver first.

[tensorkite]

Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly.

[macropoint]

Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly.

Ran a quick check on my box. The issue went away once I changed one setting and restarted the stack cleanly.