running local models on a phone

[cozysmith]

Did a bit of testing and thought I'd post the result here.

Short version: llama.cpp behaved well for me, but only after I stopped ignoring KV cache pressure.

Happy to post more details if anyone wants them.

[fusedriver]

worth checking the driver first. Turns out 'just one more test' was a lie.

[rocmstack25]

that fixed it for me too.

[batchpeak]

worth checking the driver first.

[routeline23]

My first guess would be this: On 4090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.

[batchpeak]

Ran a quick check on my box. On 5090 with Ollama, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.

[torchkite]

following this because I'm in the same boat.

[relaysupply90]

My first guess would be this: If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.

[llmpeak]

Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly.

[llm_wren]

Good point. That matches what I saw on Threadripper box. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.