recipes for a good local stack

[mlxgarden]

Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal.

[rocmcrate26]

Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal.

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at long context. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[rasterbench]

Thanks, that helps. I ended up in about the same place using llama.cpp.

[relayforge]

I tested this tonight. On M3 Max with llama.cpp, I saw basically the same behavior.

[mlxgarden]

I don't think that's the whole story. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. At this point the real bottleneck is my patience.

[mangoquill]

worth checking the driver first.