+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Edge AI & Gadgets (https://www.datasculpting.ai/inference/forumdisplay.php?fid=21)
+--- Forum: Mobile (https://www.datasculpting.ai/inference/forumdisplay.php?fid=73)
+--- Thread: on-device inference worth the effort (/showthread.php?tid=187)
Pages:
1
2
on-device inference worth the effort - marbleyard32 - 09-09-2026
This keeps coming up in conversation so I figured I'd start a thread.
The more I work with local setups, the more it seems like tokens/sec decides everything downstream.
Curious whether that lines up with what other people are seeing.
My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.
Re: on-device inference worth the effort - vramarch - 09-09-2026
I tested this tonight. On dual 3090 with TensorRT-LLM, I saw basically the same behavior. I expected the model choice to matter more than it did.
Re: on-device inference worth the effort - batchpeak - 09-12-2026
Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.
Re: on-device inference worth the effort - plainline - 09-13-2026
I'd approach it in a boring order. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: on-device inference worth the effort - routeline23 - 09-16-2026
Quote:I'd approach it in a boring order. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious. A lo
A couple questions before I guess too hard. Are you testing this on 4090 or something else?
I'm happy to test one more variant if you want a second data point.
Re: on-device inference worth the effort - sparrowghost - 09-16-2026
What exact setup are you running? Is this local-only or are you routing anything externally?
Re: on-device inference worth the effort - turbosupply83 - 09-16-2026
Quote:What exact setup are you running? Is this local-only or are you routing anything externally?
Did a small comparison this afternoon. On 5090 with SGLang, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.
Re: on-device inference worth the effort - routeline23 - 09-16-2026
Quote:Quote:What exact setup are you running? Is this local-only or are you routing anything externally?
I hit almost the same issue last night.
If you post your exact settings, people here can probably narrow it down pretty fast.
Re: on-device inference worth the effort - sparrowghost - 09-19-2026
Quote:Quote:Quote:What exact setup are you running? Is this local-only or are you routing anything externally?
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at embeddings. Not glamorous advice, but repeatable baselines save a lot of time.
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.
Re: on-device inference worth the effort - plainscope50 - 09-19-2026
A couple things I'd check first. If you're on Ollama, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.