privacy wins of local models: setup advice

[kernelgrid]

Has anyone else run into this?

I'm mostly working with Kimi in llama.cpp on A6000. The part I'm trying to understand is INT4.

Would appreciate practical advice rather than theory.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

[rocmriley]

A couple things I'd check first. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.

[sparrowworks88]

I tested this tonight. My result was mixed: first run looked bad, second run looked normal.

[mlxgarden]

Good point. I ended up in about the same place using PyTorch.

[audionode8]

Good point. The pattern around reranking has been pretty similar for me.

[llmpeak]

following this because I'm in the same boat. The logs were somehow calmer than I was.

[vramstream]

I don't think that's the whole story. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.

[nanopatch]

Can you fill in a bit more context? Is this local-only or are you routing anything externally?

[deltagrid]

I'd start with the simpler stack.

[rocmhouse90]

Interesting, I hadn't tried that. That matches what I saw on B200. The biggest change for me was simplifying the stack instead of adding one more optimization layer.