+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Security, Privacy & Safety (https://www.datasculpting.ai/inference/forumdisplay.php?fid=23)
+--- Forum: Local Privacy (https://www.datasculpting.ai/inference/forumdisplay.php?fid=76)
+--- Thread: privacy wins of local models: what are you seeing? (/showthread.php?tid=491)
Pages:
1
2
privacy wins of local models: what are you seeing? - mangoquill - 09-25-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
Just sharing a setup that ended up working reasonably well: Llama, Ollama, 4090, with a lot of attention on quant drift.
Maybe useful as a starting point for newer folks.
Re: privacy wins of local models: what are you seeing? - mlxgarden - 09-26-2026
that fixed it for me too.
Re: privacy wins of local models: what are you seeing? - vramstream - 09-26-2026
Quote:that fixed it for me too.
That matches what I was seeing. I ended up in about the same place using PyTorch. Turns out 'just one more test' was a lie.
Re: privacy wins of local models: what are you seeing? - rocmriley - 09-26-2026
Thanks, that helps. The pattern around small local stacks has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: privacy wins of local models: what are you seeing? - rocmhouse90 - 09-27-2026
A couple questions before I guess too hard. Are you testing this on 4090 or something else?
Re: privacy wins of local models: what are you seeing? - mlxgarden - 09-30-2026
What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tokens/sec.
Re: privacy wins of local models: what are you seeing? - kernelgrid - 10-01-2026
Interesting, I hadn't tried that. I ended up in about the same place using SGLang. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: privacy wins of local models: what are you seeing? - audioriver - 10-01-2026
I think the bottleneck is somewhere else. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet. Turns out 'just one more test' was a lie.
Re: privacy wins of local models: what are you seeing? - vramstream - 10-02-2026
Quote:I think the bottleneck is somewhere else. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Not saying yo
I hit almost the same issue last night.
Re: privacy wins of local models: what are you seeing? - mintstation99 - 10-02-2026
Not sure I agree there. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.