what made local AI click for you

[thetagrid15]

Maybe, but I had the opposite result. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[nanopatch]

Did a small comparison this afternoon. On 7900 XTX with MLX, I saw basically the same behavior.

[quiettoken]

Good point. The pattern around INT4 has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[tensorkite]

following this because I'm in the same boat.

[thetagrid15]

that fixed it for me too.

[batchshed]

that fixed it for me too.

Appreciate the detail. The pattern around consumer GPUs has been pretty similar for me.