+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Drivers / Frameworks / Toolkits (https://www.datasculpting.ai/inference/forumdisplay.php?fid=8)
+--- Forum: AMD / ROCm (https://www.datasculpting.ai/inference/forumdisplay.php?fid=32)
+--- Thread: ROCm version that finally behaves (/showthread.php?tid=905)
Pages:
1
2
ROCm version that finally behaves - thetaember38 - 10-08-2026
Did a bit of testing and thought I'd post the result here.
Short version: TensorRT-LLM behaved well for me, but only after I stopped ignoring tokens/sec.
Happy to post more details if anyone wants them.
Re: ROCm version that finally behaves - benchplease - 10-10-2026
Maybe, but I had the opposite result. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.
Re: ROCm version that finally behaves - rasterpoint - 10-11-2026
Quote:Maybe, but I had the opposite result. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.
That matches what I was seeing. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: ROCm version that finally behaves - batchyard - 10-11-2026
What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tokens/sec. Once you know the plain path works, then add the faster path back in piece by piece.
Re: ROCm version that finally behaves - cozykite - 10-11-2026
following this because I'm in the same boat.
Re: ROCm version that finally behaves - northhall88 - 10-12-2026
Interesting, I hadn't tried that. The pattern around INT4 has been pretty similar for me.
Re: ROCm version that finally behaves - papermixer4 - 10-13-2026
Interesting, I hadn't tried that. I ended up in about the same place using SGLang. Turns out 'just one more test' was a lie.
Re: ROCm version that finally behaves - rocmriley - 10-13-2026
What exact setup are you running? Are you testing this on B200 or something else?
Re: ROCm version that finally behaves - mintquill - 10-14-2026
Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality. I keep promising myself this is the last rebuild.
Re: ROCm version that finally behaves - rocmriley - 10-14-2026
following this because I'm in the same boat.