+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Edge AI & Gadgets (https://www.datasculpting.ai/inference/forumdisplay.php?fid=21)
+--- Forum: Mobile (https://www.datasculpting.ai/inference/forumdisplay.php?fid=73)
+--- Thread: on-device inference worth the effort (/showthread.php?tid=879)
Pages:
1
2
on-device inference worth the effort - cozysmith - 10-07-2026
This keeps coming up in conversation so I figured I'd start a thread.
The more I work with local setups, the more it seems like prefill/decode split decides everything downstream.
Maybe obvious to some of you, but it clicked for me this week.
Re: on-device inference worth the effort - sparrowshed - 10-08-2026
Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal.
Re: on-device inference worth the effort - fastquill74 - 10-08-2026
Thanks, that helps. That matches what I saw on mini PC. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.
Re: on-device inference worth the effort - amberrunner - 10-09-2026
I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.
Re: on-device inference worth the effort - macrotinker - 10-09-2026
I think the bottleneck is somewhere else. I've seen people blame tokens/sec when the real issue was much more mundane. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.
If you post your exact settings, people here can probably narrow it down pretty fast.
Re: on-device inference worth the effort - marbleyard32 - 10-09-2026
I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Could be a workload difference, but I wouldn't generalize too fast from one setup.
Re: on-device inference worth the effort - turbosupply83 - 10-09-2026
My first guess would be this: If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: on-device inference worth the effort - cachelab - 10-10-2026
Need a little more detail here. Are you testing this on 7900 XTX or something else?
Re: on-device inference worth the effort - fastquill74 - 10-12-2026
A couple questions before I guess too hard. Is this local-only or are you routing anything externally?
Re: on-device inference worth the effort - localcraft17 - 10-12-2026
Need a little more detail here. Are you testing this on A6000 or something else?