+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Hardware (https://www.datasculpting.ai/inference/forumdisplay.php?fid=4)
+--- Forum: GPUs (https://www.datasculpting.ai/inference/forumdisplay.php?fid=28)
+--- Thread: dual GPU split inference (/showthread.php?tid=208)
Pages:
1
2
dual GPU split inference - rocmriley - 09-10-2026
Spent some time on this today and wanted to compare notes.
I've gone back and forth between MLX and vLLM for FLUX. The interesting difference for me has been quant drift.
What would you pick if you had to live with it for a few months?
I care more about stability than squeezing the last possible tokens/sec out of it.
Re: dual GPU split inference - batchwire - 09-10-2026
I'd approach it in a boring order. On A6000, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Re: dual GPU split inference - minthall - 09-11-2026
Thanks, that helps. That matches what I saw on 4090.
Re: dual GPU split inference - batchember - 09-11-2026
Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?
Re: dual GPU split inference - cacheline - 09-13-2026
A couple things I'd check first. On GB200, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: dual GPU split inference - cudapoint - 09-16-2026
Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.
Re: dual GPU split inference - signaltrack44 - 09-17-2026
Quote:Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure show
I'd start with the simpler stack.
Re: dual GPU split inference - thetarunner - 09-17-2026
My first guess would be this: Before rebuilding anything, I'd test one clean baseline with FLUX and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: dual GPU split inference - sanepeak48 - 09-17-2026
A couple questions before I guess too hard. Are you testing this on GB200 or something else?
Re: dual GPU split inference - sanevault - 09-17-2026
What worked for me was pretty simple. On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.