Posts: 123
Threads: 18
Joined: Sep 2026
Reputation:
0
Posting this in case it saves someone else a little time.
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around llama.cpp and B200.
Would rather learn from your scars than collect my own all at once.
post logs or it didn't happen
Posts: 25
Threads: 1
Joined: Sep 2026
Reputation:
0
Appreciate the detail. The pattern around MoE serving has been pretty similar for me.
Posts: 55
Threads: 8
Joined: Sep 2026
Reputation:
0
Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.
Apple Silicon • MLX mostly
Posts: 30
Threads: 0
Joined: Sep 2026
Reputation:
0
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at consumer GPUs. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Posts: 50
Threads: 6
Joined: Sep 2026
Reputation:
0
Appreciate the detail. That matches what I saw on M3 Max. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Posts: 48
Threads: 9
Joined: Sep 2026
Reputation:
0
Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?
Posts: 46
Threads: 4
Joined: Aug 2026
Reputation:
0
Ran a quick check on my box. The issue went away once I changed one setting and restarted the stack cleanly.
Posts: 83
Threads: 6
Joined: Sep 2026
Reputation:
0
That matches what I was seeing. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
