what made local AI click for you

[rocmrunner40]

Did a bit of testing and thought I'd post the result here.

After a few runs on mini PC, I ended up with decent performance, but the setup only really made sense once I adjusted around MoE serving.

Happy to post more details if anyone wants them.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

[rocmplane]

What exact setup are you running? Are you testing this on GB200 or something else?

[audiorunner74]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tail latency. Once you know the plain path works, then add the faster path back in piece by piece.

[ggufwire22]

What exact setup are you running? What model, what runtime, and roughly what context size?

[batchsize]

What exact setup are you running? What model, what runtime, and roughly what context size?

[tensorkite]

My first guess would be this: If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious.

[rocmplane]

Appreciate the detail. The pattern around small local stacks has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[quiettoken]

I'd approach it in a boring order. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.

[vectorshed43]

Good point. The pattern around VRAM headroom has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[tensorkite]

That matches what I was seeing. I don't think your conclusion is far off.