CLI helpers for local models with MLX

[batchshed]

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like VRAM headroom decides everything downstream.

Curious whether that lines up with what other people are seeing.

[cachebench]

Appreciate the detail. The pattern around tool calling has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[audionode8]

A couple things I'd check first. On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[mlxgarden]

worth checking the driver first.

[rocmriley]

worth checking the driver first.

[vectorport]

My first guess would be this: On A6000, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

[kernelgrid]

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Kimi and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.

[rocmhouse90]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at token cost. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[audionode8]

following this because I'm in the same boat.

[deltagrid]

Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the simpler explanation before adopting the more interesting one.