best local open-weight model this week

[quantghost]

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: Ollama on mini PC, keeping prefix caching in mind from the start.

Maybe useful as a starting point for newer folks.

This is mostly for local work rather than public serving.

[sanevault]

I don't think that's the whole story. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[rocmrunner40]

I'd start with the simpler stack.

[thetagrid15]

What worked for me was pretty simple. On 7900 XTX, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.

[oxidemixer]

Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.

[southtinker]

I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[sanevault]

My first guess would be this: If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.

[mangohouse]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Phi and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.

[cudaworks]

A couple questions before I guess too hard. Are you testing this on B200 or something else?

[rocmrunner40]

What exact setup are you running? Are you testing this on dual 3090 or something else?

I'm happy to test one more variant if you want a second data point.