recipes for a good local stack

[vramstream]

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of MLX and Threadripper box felt surprisingly usable once I tuned around quant drift.

Maybe useful as a starting point for newer folks.

[mlxgarden]

Thought I'd post a small recipe because a lot of threads ask similar things.

What exact setup are you running? Are you testing this on 4090 or something else?

[mangoquill]

A couple questions before I guess too hard. What model, what runtime, and roughly what context size?

[rocmmixer]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at consumer GPUs. Not glamorous advice, but repeatable baselines save a lot of time.

[vramstream]

Appreciate the detail. I don't think your conclusion is far off.

[audioriver]

Need a little more detail here. Are you testing this on Threadripper box or something else?

[batchember]

Need a little more detail here. Are you testing this on Threadripper box or something else?

following this because I'm in the same boat.

[batchshed]

What worked for me was pretty simple. If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious.

[cachebench]

What worked for me was pretty simple. On dual 3090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.

[vramscope]

same here on my side.