ROCm version that finally behaves

[thetaember38]

Did a bit of testing and thought I'd post the result here.

Short version: TensorRT-LLM behaved well for me, but only after I stopped ignoring tokens/sec.

Happy to post more details if anyone wants them.

[benchplease]

Maybe, but I had the opposite result. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.

[rasterpoint]

Maybe, but I had the opposite result. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.

That matches what I was seeing. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[batchyard]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tokens/sec. Once you know the plain path works, then add the faster path back in piece by piece.

[cozykite]

following this because I'm in the same boat.

[northhall88]

Interesting, I hadn't tried that. The pattern around INT4 has been pretty similar for me.

[papermixer4]

Interesting, I hadn't tried that. I ended up in about the same place using SGLang. Turns out 'just one more test' was a lie.

[rocmriley]

What exact setup are you running? Are you testing this on B200 or something else?

[mintquill]

Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality. I keep promising myself this is the last rebuild.

[rocmriley]

following this because I'm in the same boat.