M3 Max for local use for DeepSeek

[thetaarch43]

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: llama.cpp on mini PC, keeping batch size in mind from the start.

If someone wants the exact settings I can clean them up and post them.

[mangoquill]

following this because I'm in the same boat.

[circuitwire58]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Gemma and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[thetaarch43]

that fixed it for me too.

[dockridge]

I'd approach it in a boring order. On 7900 XTX, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[quiettoken]

Good point. The pattern around token cost has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[homelabhenry]

I think the bottleneck is somewhere else. I get why that seems right, but on GB200 the bottleneck felt elsewhere. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[dockridge]

My first guess would be this: On A6000, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[mlxgarden]

Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.

[torchshed]

Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.

I don't think that's the whole story. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.