M3 Max for local use on 4090

[turboyard94]

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: PyTorch on Threadripper box, keeping small local stacks in mind from the start.

Maybe useful as a starting point for newer folks.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

[turboyard94]

I'd start with the simpler stack.

[mlxgarden]

I don't think that's the whole story. I've seen people blame router policy when the real issue was much more mundane.

[visionyard23]

A couple things I'd check first. On B200, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[mellowstream]

same here on my side.

[pixelcrate82]

A couple things I'd check first. On B200, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[mlxgarden]

Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher.

[amberplane]

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with DeepSeek and compare from there.

[turboyard94]

that fixed it for me too.

[mlxgarden]

Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal.