+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Hardware (https://www.datasculpting.ai/inference/forumdisplay.php?fid=4)
+--- Forum: GPUs (https://www.datasculpting.ai/inference/forumdisplay.php?fid=28)
+--- Thread: dual GPU split inference for Mistral (/showthread.php?tid=63)
Pages:
1
2
dual GPU split inference for Mistral - paperrunner - 08-29-2026
Posting this in case it saves someone else a little time.
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about MoE serving.
Simple and boring is a valid answer.
This is mostly for local work rather than public serving.
Re: dual GPU split inference for Mistral - turbogrid11 - 08-31-2026
Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal.
Re: dual GPU split inference for Mistral - cudapoint - 08-31-2026
Quote:Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal.
That matches what I was seeing. That matches what I saw on 5090. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: dual GPU split inference for Mistral - batchwire - 09-01-2026
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at KV cache pressure.
Re: dual GPU split inference for Mistral - turboyard94 - 09-02-2026
Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?
Re: dual GPU split inference for Mistral - dockkite69 - 09-04-2026
Quote:Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?
Need a little more detail here. Do you see the same thing with a smaller batch or shorter prompt?
Re: dual GPU split inference for Mistral - rocmriley - 09-04-2026
Good point. That matches what I saw on M3 Max. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: dual GPU split inference for Mistral - batchwire - 09-04-2026
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefix caching.
Re: dual GPU split inference for Mistral - signaltrack44 - 09-07-2026
Need a little more detail here. Is this local-only or are you routing anything externally?
Re: dual GPU split inference for Mistral - amberhall - 09-07-2026
worth checking the driver first.