+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: LLM Models (https://www.datasculpting.ai/inference/forumdisplay.php?fid=12)
+--- Forum: Open Weight (https://www.datasculpting.ai/inference/forumdisplay.php?fid=49)
+---- Forum: Local (https://www.datasculpting.ai/inference/forumdisplay.php?fid=51)
+---- Thread: recipes for a good local stack (/showthread.php?tid=720)
Pages:
1
2
recipes for a good local stack - vramstream - 10-04-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
This isn't revolutionary, but the combination of MLX and Threadripper box felt surprisingly usable once I tuned around quant drift.
Maybe useful as a starting point for newer folks.
Re: recipes for a good local stack - mlxgarden - 10-05-2026
Quote:Thought I'd post a small recipe because a lot of threads ask similar things.
What exact setup are you running? Are you testing this on 4090 or something else?
Re: recipes for a good local stack - mangoquill - 10-05-2026
A couple questions before I guess too hard. What model, what runtime, and roughly what context size?
Re: recipes for a good local stack - rocmmixer - 10-05-2026
I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at consumer GPUs. Not glamorous advice, but repeatable baselines save a lot of time.
Re: recipes for a good local stack - vramstream - 10-05-2026
Appreciate the detail. I don't think your conclusion is far off.
Re: recipes for a good local stack - audioriver - 10-05-2026
Need a little more detail here. Are you testing this on Threadripper box or something else?
Re: recipes for a good local stack - batchember - 10-05-2026
Quote:Need a little more detail here. Are you testing this on Threadripper box or something else?
following this because I'm in the same boat.
Re: recipes for a good local stack - batchshed - 10-05-2026
What worked for me was pretty simple. If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious.
Re: recipes for a good local stack - cachebench - 10-05-2026
What worked for me was pretty simple. On dual 3090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.
Re: recipes for a good local stack - vramscope - 10-05-2026
same here on my side.