running local models on a phone

[rocmplane]

A couple questions before I guess too hard. What model, what runtime, and roughly what context size?

[routeline23]

Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.

If you post your exact settings, people here can probably narrow it down pretty fast.