+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Miscellaneous Discussion (https://www.datasculpting.ai/inference/forumdisplay.php?fid=24)
+--- Forum: General (https://www.datasculpting.ai/inference/forumdisplay.php?fid=25)
+--- Thread: what made local AI click for you (/showthread.php?tid=240)
what made local AI click for you - rocmrunner40 - 09-12-2026
Did a bit of testing and thought I'd post the result here.
After a few runs on mini PC, I ended up with decent performance, but the setup only really made sense once I adjusted around MoE serving.
Happy to post more details if anyone wants them.
My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.
Re: what made local AI click for you - rocmplane - 09-14-2026
What exact setup are you running? Are you testing this on GB200 or something else?
Re: what made local AI click for you - audiorunner74 - 09-14-2026
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tail latency. Once you know the plain path works, then add the faster path back in piece by piece.
Re: what made local AI click for you - ggufwire22 - 09-14-2026
What exact setup are you running? What model, what runtime, and roughly what context size?
Re: what made local AI click for you - batchsize - 09-14-2026
What exact setup are you running? What model, what runtime, and roughly what context size?
Re: what made local AI click for you - tensorkite - 09-14-2026
My first guess would be this: If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious.
Re: what made local AI click for you - rocmplane - 09-14-2026
Appreciate the detail. The pattern around small local stacks has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: what made local AI click for you - quiettoken - 09-14-2026
I'd approach it in a boring order. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.
Re: what made local AI click for you - vectorshed43 - 09-17-2026
Good point. The pattern around VRAM headroom has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: what made local AI click for you - tensorkite - 09-18-2026
That matches what I was seeing. I don't think your conclusion is far off.