Posts: 73
Threads: 5
Joined: Aug 2026
Reputation:
0
Wanted to compare notes with people doing similar things.
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling MoE serving.
Not looking for the perfect answer, mostly what worked in real use.
Posts: 55
Threads: 8
Joined: Sep 2026
Reputation:
0
same here on my side.
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.
Apple Silicon • MLX mostly
Posts: 43
Threads: 3
Joined: Sep 2026
Reputation:
0
Appreciate the detail. I ended up in about the same place using PyTorch. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
mostly local
Posts: 168
Threads: 14
Joined: Sep 2026
Reputation:
0
Ran a quick check on my box. On 5090 with llama.cpp, I saw basically the same behavior. I expected the model choice to matter more than it did. At this point the real bottleneck is my patience.
2x3090 • llama.cpp • too much VRAM is still not enough
Posts: 41
Threads: 0
Joined: Oct 2026
Reputation:
0
My first guess would be this: On B200, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Posts: 73
Threads: 5
Joined: Aug 2026
Reputation:
0
What worked for me was pretty simple. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious.
Posts: 168
Threads: 14
Joined: Sep 2026
Reputation:
0
Need a little more detail here. What model, what runtime, and roughly what context size?
2x3090 • llama.cpp • too much VRAM is still not enough
Posts: 41
Threads: 0
Joined: Oct 2026
Reputation:
0
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at router policy.
Posts: 48
Threads: 4
Joined: Aug 2026
Reputation:
0
Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.
