+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Edge AI & Gadgets (https://www.datasculpting.ai/inference/forumdisplay.php?fid=21)
+--- Forum: AI Devices (https://www.datasculpting.ai/inference/forumdisplay.php?fid=75)
+--- Thread: dedicated AI gadgets you actually use (/showthread.php?tid=959)
Pages:
1
2
dedicated AI gadgets you actually use - flintriver50 - 10-10-2026
Posting this in case it saves someone else a little time.
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?
I'm especially interested in mistakes worth avoiding.
Re: dedicated AI gadgets you actually use - flintriver50 - 10-12-2026
Quote:Posting this in case it saves someone else a little time.
Need a little more detail here. Is this local-only or are you routing anything externally?
Re: dedicated AI gadgets you actually use - audiorunner74 - 10-15-2026
Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.
I'm happy to test one more variant if you want a second data point.
Re: dedicated AI gadgets you actually use - quantghost - 10-15-2026
I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. I expected the model choice to matter more than it did.
Re: dedicated AI gadgets you actually use - latentscope35 - 10-17-2026
Tried to reproduce it just now. On 4090 with MLX, I saw basically the same behavior. The main thing that moved was latency, not quality.
Re: dedicated AI gadgets you actually use - flintriver50 - 10-17-2026
Need a little more detail here. Do you see the same thing with a smaller batch or shorter prompt?
Re: dedicated AI gadgets you actually use - turbosupply83 - 10-17-2026
Ran a quick check on my box. On mini PC with Ollama, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.
Re: dedicated AI gadgets you actually use - routecrate - 10-18-2026
I think the bottleneck is somewhere else. I've seen people blame MoE serving when the real issue was much more mundane.
Re: dedicated AI gadgets you actually use - mintquill - 10-18-2026
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at KV cache pressure. Not glamorous advice, but repeatable baselines save a lot of time.
Re: dedicated AI gadgets you actually use - audiopoint18 - 10-18-2026
Not sure I agree there. I get why that seems right, but on 7900 XTX the bottleneck felt elsewhere. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.