on-device inference worth the effort: what are you seeing?

[batchscope72]

Appreciate the detail. The pattern around consumer GPUs has been pretty similar for me.

[fusedriver]

Can you fill in a bit more context? What model, what runtime, and roughly what context size?

[mellowstream]

Can you fill in a bit more context? What model, what runtime, and roughly what context size?

[relaysupply90]

Can you fill in a bit more context? What model, what runtime, and roughly what context size?

Appreciate the detail. That matches what I saw on 7900 XTX. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[cozysmith]

Can you fill in a bit more context? Are you testing this on GB200 or something else?

[thetaember38]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Qwen and compare from there.