ROCm version that finally behaves with MLX

[tensorworks]

Spent some time on this today and wanted to compare notes.

Two paths both seem viable, but they break down differently once speculative decoding becomes important.

Trying to avoid rebuilding the whole thing twice.

This is mostly for local work rather than public serving.

[relaynest82]

Spent some time on this today and wanted to compare notes.

I think the bottleneck is somewhere else. I get why that seems right, but on dual 3090 the bottleneck felt elsewhere. Worth testing the simpler explanation before adopting the more interesting one.

[batchsize]

I tested this tonight. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.

I'm happy to test one more variant if you want a second data point.

[dockworks23]

worth checking the driver first.

[rocmmixer]

following this because I'm in the same boat. At this point the real bottleneck is my patience.

[tinycontext]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Mixtral and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.

[batchyard]

A couple things I'd check first. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.

[tensorworks]

A couple things I'd check first. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Onc

Interesting, I hadn't tried that. I ended up in about the same place using MLX. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[rocmmixer]

[quote]A couple things I'd check first. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvio

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with FLUX and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.

[batchyard]

That matches what I was seeing. The pattern around workstation thermals has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.