+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: LLM Models (https://www.datasculpting.ai/inference/forumdisplay.php?fid=12)
+--- Forum: Open Weight (https://www.datasculpting.ai/inference/forumdisplay.php?fid=49)
+---- Forum: Local (https://www.datasculpting.ai/inference/forumdisplay.php?fid=51)
+---- Thread: recipes for a good local stack (/showthread.php?tid=285)
Pages:
1
2
recipes for a good local stack - vectorport - 09-15-2026
Spent some time on this today and wanted to compare notes.
I'm comparing two approaches here: keep it simple on 4090, or squeeze more out of the stack and deal with the complexity.
Trying to avoid rebuilding the whole thing twice.
I care more about stability than squeezing the last possible tokens/sec out of it.
Re: recipes for a good local stack - vectorport - 09-15-2026
I'd approach it in a boring order. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.
Re: recipes for a good local stack - mlxgarden - 09-16-2026
Quote:I'd approach it in a boring order. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious. Once
That matches what I was seeing. The pattern around KV cache pressure has been pretty similar for me.
Re: recipes for a good local stack - rocmriley - 09-16-2026
Maybe, but I had the opposite result. I've seen people blame INT4 when the real issue was much more mundane. Worth testing the simpler explanation before adopting the more interesting one.
Re: recipes for a good local stack - quietstack - 09-19-2026
Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.
Re: recipes for a good local stack - relayforge - 09-19-2026
Not sure I agree there. I get why that seems right, but on mini PC the bottleneck felt elsewhere.
Re: recipes for a good local stack - rocmriley - 09-20-2026
Quote:Not sure I agree there. I get why that seems right, but on mini PC the bottleneck felt elsewhere.
Appreciate the detail. That matches what I saw on Threadripper box. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: recipes for a good local stack - mlxgarden - 09-20-2026
I think the bottleneck is somewhere else. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.
Re: recipes for a good local stack - thetaplane - 09-21-2026
A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with FLUX and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.
Re: recipes for a good local stack - modgarden - 09-21-2026
Tried to reproduce it just now. On B200 with LM Studio, I saw basically the same behavior. I expected the model choice to matter more than it did.