+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Apps / Utilities / Tools (https://www.datasculpting.ai/inference/forumdisplay.php?fid=18)
+--- Forum: Online / Hosted (https://www.datasculpting.ai/inference/forumdisplay.php?fid=19)
+--- Thread: good hosted AI app that still respects privacy on 4090 (/showthread.php?tid=860)
Pages:
1
2
good hosted AI app that still respects privacy on 4090 - macrolab - 10-08-2026
Spent some time on this today and wanted to compare notes.
Two paths both seem viable, but they break down differently once reranking becomes important.
Interested in tradeoffs more than fan loyalty.
I care more about stability than squeezing the last possible tokens/sec out of it.
Re: good hosted AI app that still respects privacy on 4090 - batchshed - 10-10-2026
Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly.
Re: good hosted AI app that still respects privacy on 4090 - torchsmith25 - 10-10-2026
My first guess would be this: On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Re: good hosted AI app that still respects privacy on 4090 - amberplane - 10-11-2026
Quote:My first guess would be this: On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives.
What worked for me was pretty simple. On M3 Max, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Re: good hosted AI app that still respects privacy on 4090 - batchshed - 10-12-2026
Quote:[quote]My first guess would be this: On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives.[/q
I'd approach it in a boring order. If you're on MLX, I'd confirm the version first and then reduce variables until the behavior is obvious. The logs were somehow calmer than I was.
Re: good hosted AI app that still respects privacy on 4090 - macronest - 10-12-2026
Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?
Re: good hosted AI app that still respects privacy on 4090 - embershed99 - 10-12-2026
A couple questions before I guess too hard. Are you testing this on B200 or something else?
Re: good hosted AI app that still respects privacy on 4090 - vramarch - 10-12-2026
Tried to reproduce it just now. On A6000 with MLX, I saw basically the same behavior. I expected the model choice to matter more than it did.
Re: good hosted AI app that still respects privacy on 4090 - phaseshed - 10-12-2026
following this because I'm in the same boat.
Re: good hosted AI app that still respects privacy on 4090 - llmpeak - 10-15-2026
I think the bottleneck is somewhere else. I've seen people blame speculative decoding when the real issue was much more mundane.