on-device inference worth the effort: what are you seeing?

[torchdesk]

Spent some time on this today and wanted to compare notes.

I've gone back and forth between LM Studio and SGLang for Kimi. The interesting difference for me has been speculative decoding.

Interested in tradeoffs more than fan loyalty.

This is mostly for local work rather than public serving. Turns out 'just one more test' was a lie.