+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Edge AI & Gadgets (https://www.datasculpting.ai/inference/forumdisplay.php?fid=21)
+--- Forum: Mobile (https://www.datasculpting.ai/inference/forumdisplay.php?fid=73)
+--- Thread: running local models on a phone (/showthread.php?tid=223)
Pages:
1
2
running local models on a phone - cozysmith - 09-11-2026
Did a bit of testing and thought I'd post the result here.
Short version: llama.cpp behaved well for me, but only after I stopped ignoring KV cache pressure.
Happy to post more details if anyone wants them.
Re: running local models on a phone - fusedriver - 09-11-2026
worth checking the driver first. Turns out 'just one more test' was a lie.
Re: running local models on a phone - rocmstack25 - 09-12-2026
that fixed it for me too.
Re: running local models on a phone - batchpeak - 09-12-2026
worth checking the driver first.
Re: running local models on a phone - routeline23 - 09-14-2026
My first guess would be this: On 4090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.
Re: running local models on a phone - batchpeak - 09-15-2026
Ran a quick check on my box. On 5090 with Ollama, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.
Re: running local models on a phone - torchkite - 09-15-2026
following this because I'm in the same boat.
Re: running local models on a phone - relaysupply90 - 09-15-2026
My first guess would be this: If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.
Re: running local models on a phone - llmpeak - 09-15-2026
Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly.
Re: running local models on a phone - llm_wren - 09-15-2026
Good point. That matches what I saw on Threadripper box. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.