privacy wins of local models with PyTorch

[mlxgarden]

Curious what people here are seeing.

My current setup is SGLang with 7900 XTX, and I keep running into batch size.

Would appreciate practical advice rather than theory. The logs were somehow calmer than I was.

[rocmriley]

Appreciate the detail. The pattern around quant drift has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.