what changed after the inference shift with MLX

[papermixer4]

I'd approach it in a boring order. On dual 3090, the easiest win is often to back off the aggressive settings and see whether the problem survives.