CLI helpers for local models

[audionode8]

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on 7900 XTX, or squeeze more out of the stack and deal with the complexity.

Trying to avoid rebuilding the whole thing twice.

[kernelgrid]

I hit almost the same issue last night.

[llmpeak]

that fixed it for me too.

[mlxgarden]

worth checking the driver first.

[deltagrid]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at NVFP4. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[cachebench]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at NVFP4. A lot of these

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tail latency. Once you know the plain path works, then add the faster path back in piece by piece.

[mlxgarden]

I'd approach it in a boring order. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious.

[audionode8]

Interesting, I hadn't tried that. I ended up in about the same place using vLLM.

[rocmriley]

That matches what I was seeing. I ended up in about the same place using MLX. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[modgarden]

Need a little more detail here. Are you testing this on 4090 or something else?