+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: LLM Models (https://www.datasculpting.ai/inference/forumdisplay.php?fid=12)
+--- Forum: Open Weight (https://www.datasculpting.ai/inference/forumdisplay.php?fid=49)
+---- Forum: Local (https://www.datasculpting.ai/inference/forumdisplay.php?fid=51)
+---- Thread: simple local serving recipe (/showthread.php?tid=1254)
simple local serving recipe - rocmriley - 10-14-2026
Curious what people here are seeing.
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling MoE serving.
Would appreciate practical advice rather than theory.
This is mostly for local work rather than public serving.
Re: simple local serving recipe - vramstream - 10-15-2026
I'd start with the simpler stack.
Re: simple local serving recipe - mlxgarden - 10-15-2026
Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher.
Re: simple local serving recipe - rocmhouse90 - 10-15-2026
Quote:Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher.
I think the bottleneck is somewhere else. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Could be a workload difference, but I wouldn't generalize too fast from one setup.
If you post your exact settings, people here can probably narrow it down pretty fast.
Re: simple local serving recipe - rocmriley - 10-15-2026
My first guess would be this: If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.
Re: simple local serving recipe - relayforge - 10-15-2026
following this because I'm in the same boat.