privacy wins of local models: early impressions

[mlxgarden]

Posting this in case it saves someone else a little time.

If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Would rather learn from your scars than collect my own all at once.

[cachebench]

I tested this tonight. On M3 Max with PyTorch, I saw basically the same behavior. The main thing that moved was latency, not quality.

[kernelgrid]

My first guess would be this: On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

[audioriver]

A couple things I'd check first. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.

[mlxgarden]

Good point. I ended up in about the same place using PyTorch.

[deltagrid]

I'd approach it in a boring order. On dual 3090, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[rocmriley]

Can you fill in a bit more context? Are you testing this on 7900 XTX or something else?