+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Apps / Utilities / Tools (https://www.datasculpting.ai/inference/forumdisplay.php?fid=18)
+--- Forum: Local (https://www.datasculpting.ai/inference/forumdisplay.php?fid=20)
+--- Thread: desktop apps around local AI (/showthread.php?tid=610)
desktop apps around local AI - deltagrid - 10-01-2026
Curious what people here are seeing.
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling HBM constraints.
Would appreciate practical advice rather than theory.
Re: desktop apps around local AI - mintstation99 - 10-01-2026
worth checking the driver first.
Re: desktop apps around local AI - rocmcrate26 - 10-02-2026
What worked for me was pretty simple. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.
Re: desktop apps around local AI - vramstream - 10-03-2026
Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?
Re: desktop apps around local AI - batchshed - 10-03-2026
My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there. The logs were somehow calmer than I was.
Re: desktop apps around local AI - oxideworks59 - 10-03-2026
Quote:My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there. The logs were somehow calme
Thanks, that helps. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: desktop apps around local AI - vectorport - 10-03-2026
Need a little more detail here. What model, what runtime, and roughly what context size?
Re: desktop apps around local AI - audioriver - 10-03-2026
I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at KV cache pressure. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: desktop apps around local AI - localtinker - 10-03-2026
Interesting, I hadn't tried that. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.