batching changes after update on M3 Max

[cobaltspark]

I'm trying to sanity-check my setup.

I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling MoE serving.

If you've tested something similar, what did you keep and what did you stop bothering with?

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.

[sparrowkite53]

Can you fill in a bit more context? What model, what runtime, and roughly what context size?