simple local serving recipe

[rocmriley]

Curious what people here are seeing.

I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling MoE serving.

Would appreciate practical advice rather than theory.

This is mostly for local work rather than public serving.

[vramstream]

I'd start with the simpler stack.

[mlxgarden]

Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher.

[rocmhouse90]

Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher.

I think the bottleneck is somewhere else. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Could be a workload difference, but I wouldn't generalize too fast from one setup.

If you post your exact settings, people here can probably narrow it down pretty fast.

[rocmriley]

My first guess would be this: If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.

[relayforge]

following this because I'm in the same boat.