+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Edge AI & Gadgets (https://www.datasculpting.ai/inference/forumdisplay.php?fid=21)
+--- Forum: AI Devices (https://www.datasculpting.ai/inference/forumdisplay.php?fid=75)
+--- Thread: AI pin style devices but useful? on 4090 (/showthread.php?tid=903)
Pages:
1
2
AI pin style devices but useful? on 4090 - sparrowghost - 10-08-2026
Posting this in case it saves someone else a little time.
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about consumer GPUs.
Would rather learn from your scars than collect my own all at once.
If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.
Re: AI pin style devices but useful? on 4090 - mangopath4 - 10-08-2026
same here on my side.
Re: AI pin style devices but useful? on 4090 - lablab - 10-09-2026
Did a small comparison this afternoon. On 4090 with MLX, I saw basically the same behavior. I expected the model choice to matter more than it did.
Re: AI pin style devices but useful? on 4090 - oxidestream - 10-09-2026
Interesting, I hadn't tried that. I ended up in about the same place using Ollama.
Re: AI pin style devices but useful? on 4090 - audiorunner74 - 10-12-2026
Quote:Interesting, I hadn't tried that. I ended up in about the same place using Ollama.
What worked for me was pretty simple. On dual 3090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.
Re: AI pin style devices but useful? on 4090 - rasterbench - 10-12-2026
Appreciate the detail. The pattern around embeddings has been pretty similar for me.
Re: AI pin style devices but useful? on 4090 - routeline23 - 10-13-2026
Good point. I ended up in about the same place using llama.cpp. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: AI pin style devices but useful? on 4090 - rasterbench - 10-13-2026
Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?
Re: AI pin style devices but useful? on 4090 - routeline23 - 10-13-2026
I'd approach it in a boring order. If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.
Re: AI pin style devices but useful? on 4090 - rasterbench - 10-14-2026
I tested this tonight. On dual 3090 with llama.cpp, I saw basically the same behavior.