dual GPU split inference

[rocmriley]

Spent some time on this today and wanted to compare notes.

I've gone back and forth between MLX and vLLM for FLUX. The interesting difference for me has been quant drift.

What would you pick if you had to live with it for a few months?

I care more about stability than squeezing the last possible tokens/sec out of it.

[batchwire]

I'd approach it in a boring order. On A6000, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[minthall]

Thanks, that helps. That matches what I saw on 4090.

[batchember]

Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?

[cacheline]

A couple things I'd check first. On GB200, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[cudapoint]

Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.

[signaltrack44]

Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure show

I'd start with the simpler stack.

[thetarunner]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with FLUX and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[sanepeak48]

A couple questions before I guess too hard. Are you testing this on GB200 or something else?

[sanevault]

What worked for me was pretty simple. On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.