on-device inference worth the effort

[routeline23]

I'd approach it in a boring order. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious. A lo

A couple questions before I guess too hard. Are you testing this on 4090 or something else?

I'm happy to test one more variant if you want a second data point.