+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Drivers / Frameworks / Toolkits (https://www.datasculpting.ai/inference/forumdisplay.php?fid=8)
+--- Forum: AMD / ROCm (https://www.datasculpting.ai/inference/forumdisplay.php?fid=32)
+--- Thread: ROCm version that finally behaves with MLX (/showthread.php?tid=103)
Pages:
1
2
ROCm version that finally behaves with MLX - tensorworks - 09-02-2026
Spent some time on this today and wanted to compare notes.
Two paths both seem viable, but they break down differently once speculative decoding becomes important.
Trying to avoid rebuilding the whole thing twice.
This is mostly for local work rather than public serving.
Re: ROCm version that finally behaves with MLX - relaynest82 - 09-02-2026
Quote:Spent some time on this today and wanted to compare notes.
I think the bottleneck is somewhere else. I get why that seems right, but on dual 3090 the bottleneck felt elsewhere. Worth testing the simpler explanation before adopting the more interesting one.
Re: ROCm version that finally behaves with MLX - batchsize - 09-02-2026
I tested this tonight. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.
I'm happy to test one more variant if you want a second data point.
Re: ROCm version that finally behaves with MLX - dockworks23 - 09-03-2026
worth checking the driver first.
Re: ROCm version that finally behaves with MLX - rocmmixer - 09-04-2026
following this because I'm in the same boat. At this point the real bottleneck is my patience.
Re: ROCm version that finally behaves with MLX - tinycontext - 09-04-2026
I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Mixtral and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.
Re: ROCm version that finally behaves with MLX - batchyard - 09-04-2026
A couple things I'd check first. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.
Re: ROCm version that finally behaves with MLX - tensorworks - 09-04-2026
Quote:A couple things I'd check first. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Onc
Interesting, I hadn't tried that. I ended up in about the same place using MLX. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: ROCm version that finally behaves with MLX - rocmmixer - 09-04-2026
Quote:[quote]A couple things I'd check first. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvio
My first guess would be this: Before rebuilding anything, I'd test one clean baseline with FLUX and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.
Re: ROCm version that finally behaves with MLX - batchyard - 09-06-2026
That matches what I was seeing. The pattern around workstation thermals has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.