which local open model do you keep coming back to: what are you seeing?

[plaingarden18]

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of TensorRT-LLM and 5090 felt surprisingly usable once I tuned around NVFP4.

Maybe useful as a starting point for newer folks.

[tensorriver]

A couple questions before I guess too hard. What model, what runtime, and roughly what context size?

[edgequill]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with DeepSeek and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.

[bytelab]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with DeepSeek and compare from there. Not glamorous advice, but

Thanks, that helps. I ended up in about the same place using TensorRT-LLM. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[deepseeked]

Not sure I agree there. I've seen people blame continuous batching when the real issue was much more mundane.

[modbench]

I hit almost the same issue last night.

[promptfield]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at MoE serving. Not glamorous advice, but repeatable baselines save a lot of time.

[emberrunner]

Need a little more detail here. What model, what runtime, and roughly what context size?

[cozyarch]

Appreciate the detail. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[cachebench]

What worked for me was pretty simple. If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.