desktop apps around local AI

[deltagrid]

Curious what people here are seeing.

I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling HBM constraints.

Would appreciate practical advice rather than theory.

[mintstation99]

worth checking the driver first.

[rocmcrate26]

What worked for me was pretty simple. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

[vramstream]

Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?

[batchshed]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there. The logs were somehow calmer than I was.

[oxideworks59]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there. The logs were somehow calme

Thanks, that helps. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[vectorport]

Need a little more detail here. What model, what runtime, and roughly what context size?

[audioriver]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at KV cache pressure. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[localtinker]

Interesting, I hadn't tried that. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.