+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Miscellaneous Discussion (https://www.datasculpting.ai/inference/forumdisplay.php?fid=24)
+--- Forum: General (https://www.datasculpting.ai/inference/forumdisplay.php?fid=25)
+--- Thread: what made local AI click for you (/showthread.php?tid=835)
Pages:
1
2
what made local AI click for you - rocmrunner40 - 10-06-2026
Spent some time on this today and wanted to compare notes.
I'm comparing two approaches here: keep it simple on M3 Max, or squeeze more out of the stack and deal with the complexity.
Interested in tradeoffs more than fan loyalty.
My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.
The part that surprised me was how much the boring details mattered more than the model choice.
Re: what made local AI click for you - ggufwire22 - 10-07-2026
I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet. I keep promising myself this is the last rebuild.
Re: what made local AI click for you - stackarch - 10-07-2026
I hit almost the same issue last night.
Re: what made local AI click for you - quiettoken - 10-07-2026
I think the bottleneck is somewhere else. I've seen people blame prefill/decode split when the real issue was much more mundane. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.
Re: what made local AI click for you - plaingarden18 - 10-07-2026
A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Gemma and compare from there.
Re: what made local AI click for you - marbletrack - 10-07-2026
Quote:A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Gemma and compare from there.
I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.
Re: what made local AI click for you - quiettoken - 10-08-2026
Good point. The pattern around structured output has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: what made local AI click for you - vectorshed43 - 10-11-2026
I don't think that's the whole story. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Could be a workload difference, but I wouldn't generalize too fast from one setup.
Re: what made local AI click for you - tensorkite - 10-11-2026
I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Qwen and compare from there.
Re: what made local AI click for you - quiettoken - 10-11-2026
Maybe, but I had the opposite result. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Worth testing the simpler explanation before adopting the more interesting one.