mobile NPU reality check

[fastquill74]

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about quant drift.

Simple and boring is a valid answer.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

[llmpeak]

Posting this in case it saves someone else a little time.

worth checking the driver first.

[torchkite]

Posting this in case it saves someone else a little time.

I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. I expected the model choice to matter more than it did.

If you post your exact settings, people here can probably narrow it down pretty fast.

[plainscope50]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at small local stacks. Not glamorous advice, but repeatable baselines save a lot of time.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[sparrowworks88]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at small local stacks. N

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Mixtral and compare from there.

[relaysupply90]

same here on my side.

[rocmriley]

same here on my side.

I think the bottleneck is somewhere else. I get why that seems right, but on mini PC the bottleneck felt elsewhere. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[marbleyard32]

Appreciate the detail. That matches what I saw on B200. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

I'm happy to test one more variant if you want a second data point.

[routeline23]

Appreciate the detail. That matches what I saw on B200. Worth rechecking your defaults though, because one bad assumption can make the rest of the thr

subscribed, curious where this lands.

I'm happy to test one more variant if you want a second data point.

[sparrowghost]

That matches what I was seeing. I ended up in about the same place using vLLM. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.