+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Edge AI & Gadgets (https://www.datasculpting.ai/inference/forumdisplay.php?fid=21)
+--- Forum: Mobile (https://www.datasculpting.ai/inference/forumdisplay.php?fid=73)
+--- Thread: mobile NPU reality check (/showthread.php?tid=1372)
mobile NPU reality check - plainscope50 - 10-17-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
This isn't revolutionary, but the combination of llama.cpp and dual 3090 felt surprisingly usable once I tuned around prefix caching.
If someone wants the exact settings I can clean them up and post them.
The part that surprised me was how much the boring details mattered more than the model choice.
Re: mobile NPU reality check - vectorport - 10-17-2026
following this because I'm in the same boat.
Re: mobile NPU reality check - torchkite - 10-17-2026
A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?
Re: mobile NPU reality check - sparrowworks88 - 10-18-2026
Quote:A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?
What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there.
Re: mobile NPU reality check - amberrunner - 10-18-2026
What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tool calling. Not glamorous advice, but repeatable baselines save a lot of time.
Re: mobile NPU reality check - batchpeak - 10-19-2026
Interesting, I hadn't tried that. I ended up in about the same place using PyTorch. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: mobile NPU reality check - turboyard94 - 10-19-2026
Quote:Quote:A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?
Good point. I ended up in about the same place using MLX. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.