on-device inference worth the effort

[thetaplane]

Ran a quick check on my box. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.

[relaysupply90]

I don't think that's the whole story. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[cudacrate]

I don't think that's the whole story. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Coul

Appreciate the detail. That matches what I saw on A6000. The biggest change for me was simplifying the stack instead of adding one more optimization layer.