+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Apps / Utilities / Tools (https://www.datasculpting.ai/inference/forumdisplay.php?fid=18)
+--- Forum: Local (https://www.datasculpting.ai/inference/forumdisplay.php?fid=20)
+--- Thread: desktop apps around local AI (/showthread.php?tid=1079)
Pages:
1
2
desktop apps around local AI - audiorunner74 - 10-11-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
Just sharing a setup that ended up working reasonably well: Phi, llama.cpp, Threadripper box, with a lot of attention on structured output.
If someone wants the exact settings I can clean them up and post them.
My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.
The part that surprised me was how much the boring details mattered more than the model choice.
Re: desktop apps around local AI - llmpeak - 10-12-2026
Thanks, that helps. That matches what I saw on mini PC.
Re: desktop apps around local AI - mlxgarden - 10-12-2026
I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there.
Re: desktop apps around local AI - rocmhouse90 - 10-12-2026
I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tail latency. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
If you post your exact settings, people here can probably narrow it down pretty fast.
Re: desktop apps around local AI - rocmriley - 10-12-2026
I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.
Re: desktop apps around local AI - mlxgarden - 10-13-2026
I tested this tonight. On Threadripper box with MLX, I saw basically the same behavior.
Re: desktop apps around local AI - vectorport - 10-13-2026
same here on my side.
Re: desktop apps around local AI - audioriver - 10-13-2026
same here on my side.
Re: desktop apps around local AI - southden - 10-13-2026
Ran a quick check on my box. On 7900 XTX with MLX, I saw basically the same behavior. The main thing that moved was latency, not quality.
Re: desktop apps around local AI - cachebench - 10-13-2026
that fixed it for me too.