simple local serving recipe: setup advice

[mlxgarden]

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of MLX and A6000 felt surprisingly usable once I tuned around quant drift.

Maybe useful as a starting point for newer folks.

[cacheline]

Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?

[llmpeak]

Ran a quick check on my box. The issue went away once I changed one setting and restarted the stack cleanly.

[vramstream]

My first guess would be this: If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious.

[cachebench]

Can you fill in a bit more context? Is this local-only or are you routing anything externally?

[rocmriley]

that fixed it for me too.

[rasterbench]

What exact setup are you running? Do you see the same thing with a smaller batch or shorter prompt?

[mintstation99]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at embeddings. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[rocmriley]

I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[audioriver]

Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?