mobile NPU reality check

[plainscope50]

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of llama.cpp and dual 3090 felt surprisingly usable once I tuned around prefix caching.

If someone wants the exact settings I can clean them up and post them.

The part that surprised me was how much the boring details mattered more than the model choice.

[vectorport]

following this because I'm in the same boat.

[torchkite]

A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?

[sparrowworks88]

A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there.

[amberrunner]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tool calling. Not glamorous advice, but repeatable baselines save a lot of time.

[batchpeak]

Interesting, I hadn't tried that. I ended up in about the same place using PyTorch. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[turboyard94]

A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?

Good point. I ended up in about the same place using MLX. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.