Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on M3 Max, or squeeze more out of the stack and deal with the complexity.

Interested in tradeoffs more than fan loyalty.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

Did a bit of testing and thought I'd post the result here.

Short version: PyTorch behaved well for me, but only after I stopped ignoring NVFP4.

Curious if anyone else landed in the same place.

Has anyone else run into this?

I'm mostly working with DeepSeek in LM Studio on GB200. The part I'm trying to understand is KV cache pressure.

If you've tested something similar, what did you keep and what did you stop bothering with?

This is mostly for local work rather than public serving.

Did a bit of testing and thought I'd post the result here.

I tried this with Mistral using MLX on mini PC. The headline result was better than I expected, but token cost still ended up being the limiting factor.

Nothing scientific here, just practical notes from a real setup.

Did a bit of testing and thought I'd post the result here.

I tried this with Mixtral using MLX on B200. The headline result was better than I expected, but prefill/decode split still ended up being the limiting factor.

Nothing scientific here, just practical notes from a real setup.

I care more about stability than squeezing the last possible tokens/sec out of it.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: MLX on 7900 XTX, keeping HBM constraints in mind from the start.

If someone wants the exact settings I can clean them up and post them.

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about speculative decoding.

I'm especially interested in mistakes worth avoiding.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Llama, llama.cpp, 5090, with a lot of attention on tool calling.

Feel free to tear it apart if you see something dumb.

I care more about stability than squeezing the last possible tokens/sec out of it.

This keeps coming up in conversation so I figured I'd start a thread.

It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with INT4.

Maybe obvious to some of you, but it clicked for me this week.

Posting this in case it saves someone else a little time.

I'm not trying to build a perfect architecture diagram. I just want a sane starting point around MLX and M3 Max.

I'm especially interested in mistakes worth avoiding.

This is mostly for local work rather than public serving.