+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: LLM Models (https://www.datasculpting.ai/inference/forumdisplay.php?fid=12)
+--- Forum: Open Weight (https://www.datasculpting.ai/inference/forumdisplay.php?fid=49)
+---- Forum: Local (https://www.datasculpting.ai/inference/forumdisplay.php?fid=51)
+---- Thread: simple local serving recipe: anyone testing this? (/showthread.php?tid=263)
Pages:
1
2
simple local serving recipe: anyone testing this? - cudapoint - 09-14-2026
Posting this in case it saves someone else a little time.
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about tail latency.
I'm especially interested in mistakes worth avoiding.
For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.
Re: simple local serving recipe: anyone testing this? - mlxgarden - 09-14-2026
A couple things I'd check first. On A6000, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Re: simple local serving recipe: anyone testing this? - rocmhouse90 - 09-14-2026
A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Mixtral and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: simple local serving recipe: anyone testing this? - audioriver - 09-14-2026
A couple things I'd check first. If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.
Re: simple local serving recipe: anyone testing this? - batchshed - 09-15-2026
Quote:A couple things I'd check first. If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you
What worked for me was pretty simple. On dual 3090, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Re: simple local serving recipe: anyone testing this? - kernelgrid - 09-18-2026
same here on my side.
Re: simple local serving recipe: anyone testing this? - rocmriley - 09-18-2026
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at batch size. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: simple local serving recipe: anyone testing this? - modgarden - 09-18-2026
Quote:My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at batch size. A lot of these th
A couple things I'd check first. On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.
Re: simple local serving recipe: anyone testing this? - rocmcrate26 - 09-18-2026
Interesting, I hadn't tried that. The pattern around prefix caching has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: simple local serving recipe: anyone testing this? - batchshed - 09-19-2026
Appreciate the detail. I don't think your conclusion is far off.