on-device inference worth the effort: what are you seeing?

[cudacrate]

This keeps coming up in conversation so I figured I'd start a thread.

Appreciate the detail. I ended up in about the same place using PyTorch. The biggest change for me was simplifying the stack instead of adding one more optimization layer.