which local open model do you keep coming back to

[batchshed]

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about continuous batching.

I'm especially interested in mistakes worth avoiding.

[batchshed]

I hit almost the same issue last night.

[mintstation99]

Need a little more detail here. Are you testing this on Threadripper box or something else?

[rocmpath]

that fixed it for me too.

[southpatch]

that fixed it for me too.

[mangovault67]

Thanks, that helps. I ended up in about the same place using Ollama. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[dockworks23]

I'd start with the simpler stack.

[emberrunner]

I'd approach it in a boring order. On A6000, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time. The logs were somehow calmer than I was.

[macrotinker]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Gemma and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.

[southspark]

I'd approach it in a boring order. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survives.