7900 XTX local stack: anyone testing this?

[cobaltworks25]

Did a bit of testing and thought I'd post the result here.

After a few runs on M3 Max, I ended up with decent performance, but the setup only really made sense once I adjusted around continuous batching.

Happy to post more details if anyone wants them.

[rocmmixer]

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Kimi and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.

[cozykite]

following this because I'm in the same boat.

[quantshed]

following this because I'm in the same boat.

What exact setup are you running? What model, what runtime, and roughly what context size?

[vectorember]

Good point. I ended up in about the same place using vLLM.

[thetaember38]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at batch size.

[torchsmith25]

Appreciate the detail. I don't think your conclusion is far off.

[rocmmixer]

Ran a quick check on my box. On M3 Max with TensorRT-LLM, I saw basically the same behavior. I expected the model choice to matter more than it did.