7900 XTX local stack

[mintquill]

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: MLX on B200, keeping token cost in mind from the start.

Maybe useful as a starting point for newer folks.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

[relaynest82]

Thought I'd post a small recipe because a lot of threads ask similar things.

That matches what I was seeing. I ended up in about the same place using vLLM. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[northhall88]

Appreciate the detail. I don't think your conclusion is far off.

[dockworks23]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with FLUX and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[mintquill]

What worked for me was pretty simple. On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it. I keep promising myself this is the last rebuild.

[tensorworks]

I think the bottleneck is somewhere else. I've seen people blame HBM constraints when the real issue was much more mundane. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[plaingarden18]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at small local stacks.

[rocmriley]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at small local stacks.

My first guess would be this: If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[tensorworks]

Good point. The pattern around quant drift has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[rocmriley]

Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher. I expected the model choice to matter more than it did.