on-device inference worth the effort: what are you seeing?

[relaysupply90]

Can you fill in a bit more context? What model, what runtime, and roughly what context size?

Appreciate the detail. That matches what I saw on 7900 XTX. The biggest change for me was simplifying the stack instead of adding one more optimization layer.