+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: LLM Models (https://www.datasculpting.ai/inference/forumdisplay.php?fid=12)
+--- Forum: Open Weight (https://www.datasculpting.ai/inference/forumdisplay.php?fid=49)
+---- Forum: Hosted (https://www.datasculpting.ai/inference/forumdisplay.php?fid=50)
+---- Thread: which local open model do you keep coming back to (/showthread.php?tid=1030)
Pages:
1
2
which local open model do you keep coming back to - batchshed - 10-11-2026
Posting this in case it saves someone else a little time.
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about continuous batching.
I'm especially interested in mistakes worth avoiding.
Re: which local open model do you keep coming back to - batchshed - 10-12-2026
I hit almost the same issue last night.
Re: which local open model do you keep coming back to - mintstation99 - 10-13-2026
Need a little more detail here. Are you testing this on Threadripper box or something else?
Re: which local open model do you keep coming back to - rocmpath - 10-13-2026
that fixed it for me too.
Re: which local open model do you keep coming back to - southpatch - 10-13-2026
that fixed it for me too.
Re: which local open model do you keep coming back to - mangovault67 - 10-13-2026
Thanks, that helps. I ended up in about the same place using Ollama. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: which local open model do you keep coming back to - dockworks23 - 10-15-2026
I'd start with the simpler stack.
Re: which local open model do you keep coming back to - emberrunner - 10-15-2026
I'd approach it in a boring order. On A6000, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time. The logs were somehow calmer than I was.
Re: which local open model do you keep coming back to - macrotinker - 10-15-2026
I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Gemma and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.
Re: which local open model do you keep coming back to - southspark - 10-16-2026
I'd approach it in a boring order. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survives.