+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Security, Privacy & Safety (https://www.datasculpting.ai/inference/forumdisplay.php?fid=23)
+--- Forum: Local Privacy (https://www.datasculpting.ai/inference/forumdisplay.php?fid=76)
+--- Thread: privacy wins of local models: setup advice (/showthread.php?tid=85)
Pages:
1
2
privacy wins of local models: setup advice - kernelgrid - 08-31-2026
Has anyone else run into this?
I'm mostly working with Kimi in llama.cpp on A6000. The part I'm trying to understand is INT4.
Would appreciate practical advice rather than theory.
My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.
Re: privacy wins of local models: setup advice - rocmriley - 09-01-2026
A couple things I'd check first. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.
Re: privacy wins of local models: setup advice - sparrowworks88 - 09-02-2026
I tested this tonight. My result was mixed: first run looked bad, second run looked normal.
Re: privacy wins of local models: setup advice - mlxgarden - 09-05-2026
Good point. I ended up in about the same place using PyTorch.
Re: privacy wins of local models: setup advice - audionode8 - 09-05-2026
Good point. The pattern around reranking has been pretty similar for me.
Re: privacy wins of local models: setup advice - llmpeak - 09-05-2026
following this because I'm in the same boat. The logs were somehow calmer than I was.
Re: privacy wins of local models: setup advice - vramstream - 09-06-2026
I don't think that's the whole story. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.
Re: privacy wins of local models: setup advice - nanopatch - 09-06-2026
Can you fill in a bit more context? Is this local-only or are you routing anything externally?
Re: privacy wins of local models: setup advice - deltagrid - 09-07-2026
I'd start with the simpler stack.
Re: privacy wins of local models: setup advice - rocmhouse90 - 09-07-2026
Interesting, I hadn't tried that. That matches what I saw on B200. The biggest change for me was simplifying the stack instead of adding one more optimization layer.