+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: LLM Models (https://www.datasculpting.ai/inference/forumdisplay.php?fid=12)
+--- Forum: Open Weight (https://www.datasculpting.ai/inference/forumdisplay.php?fid=49)
+---- Forum: Hosted (https://www.datasculpting.ai/inference/forumdisplay.php?fid=50)
+---- Thread: which local open model do you keep coming back to: what are you seeing? (/showthread.php?tid=224)
Pages:
1
2
which local open model do you keep coming back to: what are you seeing? - plaingarden18 - 09-11-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
This isn't revolutionary, but the combination of TensorRT-LLM and 5090 felt surprisingly usable once I tuned around NVFP4.
Maybe useful as a starting point for newer folks.
Re: which local open model do you keep coming back to: what are you seeing? - tensorriver - 09-11-2026
A couple questions before I guess too hard. What model, what runtime, and roughly what context size?
Re: which local open model do you keep coming back to: what are you seeing? - edgequill - 09-11-2026
My first guess would be this: Before rebuilding anything, I'd test one clean baseline with DeepSeek and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.
Re: which local open model do you keep coming back to: what are you seeing? - bytelab - 09-11-2026
Quote:My first guess would be this: Before rebuilding anything, I'd test one clean baseline with DeepSeek and compare from there. Not glamorous advice, but
Thanks, that helps. I ended up in about the same place using TensorRT-LLM. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: which local open model do you keep coming back to: what are you seeing? - deepseeked - 09-11-2026
Not sure I agree there. I've seen people blame continuous batching when the real issue was much more mundane.
Re: which local open model do you keep coming back to: what are you seeing? - modbench - 09-11-2026
I hit almost the same issue last night.
Re: which local open model do you keep coming back to: what are you seeing? - promptfield - 09-11-2026
I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at MoE serving. Not glamorous advice, but repeatable baselines save a lot of time.
Re: which local open model do you keep coming back to: what are you seeing? - emberrunner - 09-11-2026
Need a little more detail here. What model, what runtime, and roughly what context size?
Re: which local open model do you keep coming back to: what are you seeing? - cozyarch - 09-12-2026
Appreciate the detail. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: which local open model do you keep coming back to: what are you seeing? - cachebench - 09-13-2026
What worked for me was pretty simple. If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.