M3 Max for local use

[tinycontext]

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Gemma, Ollama, M3 Max, with a lot of attention on router policy.

Maybe useful as a starting point for newer folks.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.

[tensorkite]

Appreciate the detail. That matches what I saw on dual 3090.

[vectorcrate98]

Good point. The pattern around token cost has been pretty similar for me.

[quietarch]

Good point. The pattern around token cost has been pretty similar for me.

I'd approach it in a boring order. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[circuitwire58]

That matches what I was seeing. That matches what I saw on B200. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[torchshed]

same here on my side.

[labnest]

My first guess would be this: If you're on Ollama, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

I'm happy to test one more variant if you want a second data point.

[mlxgarden]

My first guess would be this: On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[sanescope]

Appreciate the detail. The pattern around structured output has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[visionyard23]

That matches what I was seeing. I ended up in about the same place using vLLM. The biggest change for me was simplifying the stack instead of adding one more optimization layer.