Did a small comparison this afternoon. On mini PC with TensorRT-LLM, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.

worth checking the driver first.

Appreciate the detail. I ended up in about the same place using vLLM. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

I hit almost the same issue last night.

Quote:Appreciate the detail. I don't think your conclusion is far off.

Interesting, I hadn't tried that. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

Interesting, I hadn't tried that. I ended up in about the same place using Ollama. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

Quote:

Quote:that fixed it for me too.

subscribed, curious where this lands.

I'm happy to test one more variant if you want a second data point.

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of LM Studio and 5090 felt surprisingly usable once I tuned around MoE serving.

Maybe useful as a starting point for newer folks.