+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: LLM Models (https://www.datasculpting.ai/inference/forumdisplay.php?fid=12)
+--- Forum: Open Weight (https://www.datasculpting.ai/inference/forumdisplay.php?fid=49)
+---- Forum: Recipes & Configurations (https://www.datasculpting.ai/inference/forumdisplay.php?fid=52)
+---- Thread: simple router policy ideas with TensorRT-LLM (/showthread.php?tid=961)
Pages:
1
2
simple router policy ideas with TensorRT-LLM - rocmpath - 10-10-2026
This keeps coming up in conversation so I figured I'd start a thread.
The more I work with local setups, the more it seems like MoE serving decides everything downstream.
Maybe obvious to some of you, but it clicked for me this week.
My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.
Re: simple router policy ideas with TensorRT-LLM - minthall - 10-12-2026
Thanks, that helps. I ended up in about the same place using Ollama.
Re: simple router policy ideas with TensorRT-LLM - kernelghost - 10-12-2026
Quote:Thanks, that helps. I ended up in about the same place using Ollama.
I'd approach it in a boring order. If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.
Re: simple router policy ideas with TensorRT-LLM - quantghost - 10-12-2026
Tried to reproduce it just now. On 5090 with LM Studio, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.
Re: simple router policy ideas with TensorRT-LLM - kernelmike - 10-15-2026
worth checking the driver first.
Re: simple router policy ideas with TensorRT-LLM - rocmhouse90 - 10-15-2026
What worked for me was pretty simple. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.
Re: simple router policy ideas with TensorRT-LLM - bytepilot - 10-15-2026
same here on my side.
Re: simple router policy ideas with TensorRT-LLM - vramstream - 10-15-2026
that fixed it for me too.
Re: simple router policy ideas with TensorRT-LLM - quietstack - 10-15-2026
Interesting, I hadn't tried that. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: simple router policy ideas with TensorRT-LLM - ggufwire22 - 10-15-2026
Thanks, that helps. I ended up in about the same place using PyTorch. The biggest change for me was simplifying the stack instead of adding one more optimization layer.