+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: LLM Models (https://www.datasculpting.ai/inference/forumdisplay.php?fid=12)
+--- Forum: Open Weight (https://www.datasculpting.ai/inference/forumdisplay.php?fid=49)
+---- Forum: Local (https://www.datasculpting.ai/inference/forumdisplay.php?fid=51)
+---- Thread: simple local serving recipe: early impressions (/showthread.php?tid=123)
simple local serving recipe: early impressions - mlxgarden - 09-03-2026
This keeps coming up in conversation so I figured I'd start a thread.
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with small local stacks.
Curious whether that lines up with what other people are seeing.
Re: simple local serving recipe: early impressions - mintstation99 - 09-03-2026
A couple questions before I guess too hard. Are you testing this on A6000 or something else?
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.
Re: simple local serving recipe: early impressions - mlxgarden - 09-04-2026
Need a little more detail here. Are you testing this on 4090 or something else?
Re: simple local serving recipe: early impressions - vramstream - 09-04-2026
Quote:Need a little more detail here. Are you testing this on 4090 or something else?
Good point. I don't think your conclusion is far off.
Re: simple local serving recipe: early impressions - cachebench - 09-04-2026
subscribed, curious where this lands.
Re: simple local serving recipe: early impressions - mlxgarden - 09-04-2026
A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?
Re: simple local serving recipe: early impressions - relayforge - 09-04-2026
What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at router policy.
Re: simple local serving recipe: early impressions - rocmhouse90 - 09-04-2026
Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.
Re: simple local serving recipe: early impressions - audionode8 - 09-05-2026
That matches what I was seeing. I don't think your conclusion is far off.
Re: simple local serving recipe: early impressions - deltagrid - 09-06-2026
Thanks, that helps. That matches what I saw on B200. The biggest change for me was simplifying the stack instead of adding one more optimization layer.