on-device inference worth the effort: what are you seeing?

[torchdesk]

Spent some time on this today and wanted to compare notes.

I've gone back and forth between LM Studio and SGLang for Kimi. The interesting difference for me has been speculative decoding.

Interested in tradeoffs more than fan loyalty.

This is mostly for local work rather than public serving. Turns out 'just one more test' was a lie.

[routeline23]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Qwen and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.

[sparrowghost]

Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.

[plainscope50]

A couple things I'd check first. On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.

[fusedriver]

A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?

[mellowwire98]

Need a little more detail here. Is this local-only or are you routing anything externally?

[batchpeak]

What exact setup are you running? Are you testing this on B200 or something else?