+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: LLM Models (https://www.datasculpting.ai/inference/forumdisplay.php?fid=12)
+--- Forum: Open Weight (https://www.datasculpting.ai/inference/forumdisplay.php?fid=49)
+---- Forum: Local (https://www.datasculpting.ai/inference/forumdisplay.php?fid=51)
+---- Thread: recipes for a good local stack (/showthread.php?tid=977)
Pages:
1
2
recipes for a good local stack - modgarden - 10-09-2026
This keeps coming up in conversation so I figured I'd start a thread.
The more I work with local setups, the more it seems like NVFP4 decides everything downstream.
Wondering if this is just where the whole space is heading now.
Re: recipes for a good local stack - kernelgrid - 10-09-2026
Quote:This keeps coming up in conversation so I figured I'd start a thread.
Appreciate the detail. That matches what I saw on 4090. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: recipes for a good local stack - quantyard - 10-10-2026
Quote:Quote:This keeps coming up in conversation so I figured I'd start a thread.
Ran a quick check on my box. On A6000 with SGLang, I saw basically the same behavior. The main thing that moved was latency, not quality.
Re: recipes for a good local stack - vectorport - 10-11-2026
Quote:Quote:Quote:This keeps coming up in conversation so I figured I'd start a thread.
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at quant drift. Once you know the plain path works, then add the faster path back in piece by piece.
Re: recipes for a good local stack - modgarden - 10-11-2026
Good point. That matches what I saw on mini PC. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: recipes for a good local stack - rocmcrate26 - 10-11-2026
Quote:Good point. That matches what I saw on mini PC. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look
Good point. I ended up in about the same place using llama.cpp. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: recipes for a good local stack - rocmriley - 10-11-2026
Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.
Re: recipes for a good local stack - relayforge - 10-12-2026
What exact setup are you running? What model, what runtime, and roughly what context size?
Re: recipes for a good local stack - tinycontext - 10-12-2026
Quote:What exact setup are you running? What model, what runtime, and roughly what context size?
I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.
Re: recipes for a good local stack - mlxgarden - 10-12-2026
same here on my side.