desktop apps around local AI

[audiorunner74]

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Phi, llama.cpp, Threadripper box, with a lot of attention on structured output.

If someone wants the exact settings I can clean them up and post them.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

The part that surprised me was how much the boring details mattered more than the model choice.

[llmpeak]

Thanks, that helps. That matches what I saw on mini PC.

[mlxgarden]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there.

[rocmhouse90]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tail latency. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

If you post your exact settings, people here can probably narrow it down pretty fast.

[rocmriley]

I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.

[mlxgarden]

I tested this tonight. On Threadripper box with MLX, I saw basically the same behavior.

[vectorport]

same here on my side.

[audioriver]

same here on my side.

[southden]

Ran a quick check on my box. On 7900 XTX with MLX, I saw basically the same behavior. The main thing that moved was latency, not quality.

[cachebench]

that fixed it for me too.