running local models on a phone with PyTorch

[benchplease]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Qwen and compare from there.

[torchkite]

Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?

[marbleyard32]

Good point. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.