M3 Max for local use for Mixtral

[visionyard23]

Wanted to compare notes with people doing similar things.

My current setup is LM Studio with 7900 XTX, and I keep running into token cost.

Not looking for the perfect answer, mostly what worked in real use.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.

[quietarch]

I think the bottleneck is somewhere else. I get why that seems right, but on A6000 the bottleneck felt elsewhere.

[turboyard94]

I think the bottleneck is somewhere else. I get why that seems right, but on A6000 the bottleneck felt elsewhere.

Good point. That matches what I saw on 7900 XTX. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[dockriver]

subscribed, curious where this lands.

[visionyard23]

Ran a quick check on my box. On dual 3090 with SGLang, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.

If you post your exact settings, people here can probably narrow it down pretty fast.

[labnest]

A couple questions before I guess too hard. Are you testing this on 5090 or something else?