+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Hardware (https://www.datasculpting.ai/inference/forumdisplay.php?fid=4)
+--- Forum: GPUs (https://www.datasculpting.ai/inference/forumdisplay.php?fid=28)
+--- Thread: dual GPU split inference (/showthread.php?tid=1485)
Pages:
1
2
Re: dual GPU split inference - dockkite69 - 10-18-2026
Interesting, I hadn't tried that. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: dual GPU split inference - turboyard94 - 10-18-2026
I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Mixtral and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.
Re: dual GPU split inference - embersmith - 10-18-2026
Interesting, I hadn't tried that. That matches what I saw on A6000. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: dual GPU split inference - flintriver50 - 10-19-2026
Quote:What exact setup are you running? Do you see the same thing with a smaller batch or shorter prompt?
Not sure I agree there. I've seen people blame structured output when the real issue was much more mundane.
Re: dual GPU split inference - rocmriley - 10-19-2026
What worked for me was pretty simple. If you're on MLX, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: dual GPU split inference - sanevault - 10-19-2026
What exact setup are you running? Do you see the same thing with a smaller batch or shorter prompt?
Re: dual GPU split inference - batchember - 10-19-2026
Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly.
dual GPU split inference - dockworks23 - 10-19-2026
Spent some time on this today and wanted to compare notes.
I'm comparing two approaches here: keep it simple on A6000, or squeeze more out of the stack and deal with the complexity.
What would you pick if you had to live with it for a few months?
My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.
Re: dual GPU split inference - torchshed - 10-19-2026
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at INT4. Not glamorous advice, but repeatable baselines save a lot of time.
Re: dual GPU split inference - rocmriley - 10-19-2026
Quote:Interesting, I hadn't tried that. That matches what I saw on A6000. I got better results once I stopped chasing one metric and just looked at whether
I'd approach it in a boring order. On M3 Max, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.