prompt length getting silly on A6000

[modgarden]

Did a bit of testing and thought I'd post the result here.

I tried this with DeepSeek using LM Studio on 5090. The headline result was better than I expected, but tokens/sec still ended up being the limiting factor.

Curious if anyone else landed in the same place.

I care more about stability than squeezing the last possible tokens/sec out of it.

[mlxgarden]

same here on my side.

[localpatch]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at consumer GPUs. Once you know the plain path works, then add the faster path back in piece by piece. Turns out 'just one more test' was a lie.

[paperport64]

A couple things I'd check first. If you're on MLX, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[thetaarch43]

Good point. I don't think your conclusion is far off.

[mintstation99]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at embeddings. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.