Posts: 52
Threads: 6
Joined: Sep 2026
Reputation:
0
Curious what people here are seeing.
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling prefill/decode split.
Not looking for the perfect answer, mostly what worked in real use.
Posts: 63
Threads: 5
Joined: Sep 2026
Reputation:
0
Good point. That matches what I saw on B200. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Posts: 137
Threads: 16
Joined: Sep 2026
Reputation:
0
Good point. I ended up in about the same place using MLX. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
throughput matters, but tail latency matters too
Posts: 137
Threads: 16
Joined: Sep 2026
Reputation:
0
Good point. The pattern around MoE serving has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
throughput matters, but tail latency matters too
Posts: 37
Threads: 2
Joined: Aug 2026
Reputation:
0
What exact setup are you running? Is this local-only or are you routing anything externally?
small models can still be good
