+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Edge AI & Gadgets (https://www.datasculpting.ai/inference/forumdisplay.php?fid=21)
+--- Forum: Mobile (https://www.datasculpting.ai/inference/forumdisplay.php?fid=73)
+--- Thread: on-device inference worth the effort: what are you seeing? (/showthread.php?tid=153)
on-device inference worth the effort: what are you seeing? - torchdesk - 09-06-2026
Spent some time on this today and wanted to compare notes.
I've gone back and forth between LM Studio and SGLang for Kimi. The interesting difference for me has been speculative decoding.
Interested in tradeoffs more than fan loyalty.
This is mostly for local work rather than public serving. Turns out 'just one more test' was a lie.
Re: on-device inference worth the effort: what are you seeing? - routeline23 - 09-06-2026
I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Qwen and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.
Re: on-device inference worth the effort: what are you seeing? - sparrowghost - 09-06-2026
Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.
Re: on-device inference worth the effort: what are you seeing? - plainscope50 - 09-06-2026
A couple things I'd check first. On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.
Re: on-device inference worth the effort: what are you seeing? - fusedriver - 09-06-2026
A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?
Re: on-device inference worth the effort: what are you seeing? - mellowwire98 - 09-06-2026
Need a little more detail here. Is this local-only or are you routing anything externally?
Re: on-device inference worth the effort: what are you seeing? - batchpeak - 09-06-2026
What exact setup are you running? Are you testing this on B200 or something else?