+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: LLM Models (https://www.datasculpting.ai/inference/forumdisplay.php?fid=12)
+--- Forum: Open Weight (https://www.datasculpting.ai/inference/forumdisplay.php?fid=49)
+---- Forum: Local (https://www.datasculpting.ai/inference/forumdisplay.php?fid=51)
+---- Thread: recipes for a good local stack (/showthread.php?tid=539)
Pages:
1
2
recipes for a good local stack - rocmriley - 09-27-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
This isn't revolutionary, but the combination of Ollama and mini PC felt surprisingly usable once I tuned around prefill/decode split.
Maybe useful as a starting point for newer folks.
For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.
Re: recipes for a good local stack - mlxgarden - 09-27-2026
Thanks, that helps. I ended up in about the same place using SGLang.
Re: recipes for a good local stack - rocmriley - 09-27-2026
I don't think that's the whole story. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.
Re: recipes for a good local stack - minthall - 09-27-2026
I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher.
Re: recipes for a good local stack - kernelarch - 09-30-2026
Good point. That matches what I saw on mini PC.
Re: recipes for a good local stack - mintquill - 10-01-2026
Did a small comparison this afternoon. On A6000 with vLLM, I saw basically the same behavior. I expected the model choice to matter more than it did.
Re: recipes for a good local stack - rocmriley - 10-04-2026
Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the simpler explanation before adopting the more interesting one.
Re: recipes for a good local stack - cobalttinker - 10-04-2026
Maybe, but I had the opposite result. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Could be a workload difference, but I wouldn't generalize too fast from one setup.
Re: recipes for a good local stack - rocmriley - 10-04-2026
Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.
Re: recipes for a good local stack - kernelghost - 10-04-2026
Appreciate the detail. I ended up in about the same place using PyTorch. The biggest change for me was simplifying the stack instead of adding one more optimization layer.