setup notes worth sharing

[vectorport]

This keeps coming up in conversation so I figured I'd start a thread.

It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with KV cache pressure.

Wondering if this is just where the whole space is heading now.

I care more about stability than squeezing the last possible tokens/sec out of it.

[mlxgarden]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at continuous batching.

[rasterbench]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at continuous batching.

Not sure I agree there. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.

[mlxgarden]

Thanks, that helps. I don't think your conclusion is far off.

[cacheworks]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tail latency. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[llmpeak]

I'd approach it in a boring order. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious.

[audioriver]

I'd approach it in a boring order. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious.

Can you fill in a bit more context? Are you testing this on M3 Max or something else?

[kernelgrid]

[quote]I'd approach it in a boring order. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obviou

What worked for me was pretty simple. If you're on MLX, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.

[audionode8]

What worked for me was pretty simple. If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious.

[fastlab]

Not sure I agree there. I've seen people blame HBM constraints when the real issue was much more mundane. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.