+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Edge AI & Gadgets (https://www.datasculpting.ai/inference/forumdisplay.php?fid=21)
+--- Forum: Mobile (https://www.datasculpting.ai/inference/forumdisplay.php?fid=73)
+--- Thread: running local models on a phone (/showthread.php?tid=1454)
Pages:
1
2
Re: running local models on a phone - fusedriver - 10-18-2026
Quote:Appreciate the detail. I don't think your conclusion is far off.
I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at quant drift.
Re: running local models on a phone - llmpeak - 10-18-2026
Appreciate the detail. I don't think your conclusion is far off.
Re: running local models on a phone - torchkite - 10-18-2026
Interesting, I hadn't tried that. I ended up in about the same place using PyTorch. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
If you post your exact settings, people here can probably narrow it down pretty fast.
Re: running local models on a phone - marbleyard32 - 10-18-2026
Quote:Interesting, I hadn't tried that. I ended up in about the same place using PyTorch. The biggest change for me was simplifying the stack instead of add
I don't think that's the whole story. I've seen people blame long context when the real issue was much more mundane. Could be a workload difference, but I wouldn't generalize too fast from one setup.
Re: running local models on a phone - dockbrook21 - 10-19-2026
Quote:Quote:Appreciate the detail. I don't think your conclusion is far off.
That matches what I was seeing. The pattern around router policy has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: running local models on a phone - sparrowghost - 10-19-2026
Quote:Quote:Quote:Appreciate the detail. I don't think your conclusion is far off.
subscribed, curious where this lands.
Re: running local models on a phone - fastquill74 - 10-19-2026
I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.
I'm happy to test one more variant if you want a second data point.
Re: running local models on a phone - phasescope - 10-19-2026
Interesting, I hadn't tried that. I ended up in about the same place using vLLM.
Re: running local models on a phone - plainscope50 - 10-19-2026
A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Kimi and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.
If you post your exact settings, people here can probably narrow it down pretty fast.
running local models on a phone - batchpeak - 10-19-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
This isn't revolutionary, but the combination of PyTorch and 7900 XTX felt surprisingly usable once I tuned around continuous batching.
Maybe useful as a starting point for newer folks.
I care more about stability than squeezing the last possible tokens/sec out of it.