running local models on a phone

[marbleyard32]

Interesting, I hadn't tried that. I ended up in about the same place using PyTorch. The biggest change for me was simplifying the stack instead of add

I don't think that's the whole story. I've seen people blame long context when the real issue was much more mundane. Could be a workload difference, but I wouldn't generalize too fast from one setup.