on-device inference worth the effort

[macrotinker]

I think the bottleneck is somewhere else. I've seen people blame tokens/sec when the real issue was much more mundane. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

If you post your exact settings, people here can probably narrow it down pretty fast.