best local open-weight model this week: what are you seeing?

[mintstation99]

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.

I'm happy to test one more variant if you want a second data point.

[deltagrid]

same here on my side.

[visionyard23]

same here on my side.

Thanks, that helps. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

If you post your exact settings, people here can probably narrow it down pretty fast.

[sysstation69]

I think the bottleneck is somewhere else. I get why that seems right, but on dual 3090 the bottleneck felt elsewhere. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

I'm happy to test one more variant if you want a second data point.

[promptpoint]

Good point. I ended up in about the same place using llama.cpp. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[batchyard]

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with DeepSeek and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.