simple router policy ideas with TensorRT-LLM

[rocmpath]

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like MoE serving decides everything downstream.

Maybe obvious to some of you, but it clicked for me this week.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

[minthall]

Thanks, that helps. I ended up in about the same place using Ollama.

[kernelghost]

Thanks, that helps. I ended up in about the same place using Ollama.

I'd approach it in a boring order. If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.

[quantghost]

Tried to reproduce it just now. On 5090 with LM Studio, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.

[kernelmike]

worth checking the driver first.

[rocmhouse90]

What worked for me was pretty simple. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[bytepilot]

same here on my side.

[vramstream]

that fixed it for me too.

[quietstack]

Interesting, I hadn't tried that. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[ggufwire22]

Thanks, that helps. I ended up in about the same place using PyTorch. The biggest change for me was simplifying the stack instead of adding one more optimization layer.