privacy wins of local models: what are you seeing?

[mangoquill]

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Llama, Ollama, 4090, with a lot of attention on quant drift.

Maybe useful as a starting point for newer folks.

[mlxgarden]

that fixed it for me too.

[vramstream]

that fixed it for me too.

That matches what I was seeing. I ended up in about the same place using PyTorch. Turns out 'just one more test' was a lie.

[rocmriley]

Thanks, that helps. The pattern around small local stacks has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[rocmhouse90]

A couple questions before I guess too hard. Are you testing this on 4090 or something else?

[mlxgarden]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tokens/sec.

[kernelgrid]

Interesting, I hadn't tried that. I ended up in about the same place using SGLang. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[audioriver]

I think the bottleneck is somewhere else. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet. Turns out 'just one more test' was a lie.

[vramstream]

I think the bottleneck is somewhere else. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Not saying yo

I hit almost the same issue last night.

[mintstation99]

Not sure I agree there. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.