desktop apps around local AI: setup advice

[rocmriley]

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on Threadripper box, or squeeze more out of the stack and deal with the complexity.

Interested in tradeoffs more than fan loyalty.

[vrampilot]

Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. I expected the model choice to matter more than it did.

[cacheworks]

Interesting, I hadn't tried that. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[rocmriley]

Interesting, I hadn't tried that. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one

following this because I'm in the same boat.

[mlxgarden]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with FLUX and compare from there.

[vramstream]

Can you fill in a bit more context? What model, what runtime, and roughly what context size?

[cachebench]

A couple things I'd check first. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it. Turns out 'just one more test' was a lie.

[bytegarden73]

Thanks, that helps. I ended up in about the same place using TensorRT-LLM. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[mlxgarden]

A couple things I'd check first. If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious.

[rocmriley]

I'd approach it in a boring order. If you're on Ollama, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it. At this point the real bottleneck is my patience.