on-device inference worth the effort

[cozysmith]

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like prefill/decode split decides everything downstream.

Maybe obvious to some of you, but it clicked for me this week.