+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Generative AI (https://www.datasculpting.ai/inference/forumdisplay.php?fid=17)
+--- Forum: Multimodal (https://www.datasculpting.ai/inference/forumdisplay.php?fid=72)
+--- Thread: Multimodal discussion on GB200 (/showthread.php?tid=844)
Pages:
1
2
Multimodal discussion on GB200 - southtinker - 10-07-2026
I'm trying to sanity-check my setup.
I'm mostly working with Mistral in MLX on 5090. The part I'm trying to understand is continuous batching.
Not looking for the perfect answer, mostly what worked in real use.
I care more about stability than squeezing the last possible tokens/sec out of it.
Re: Multimodal discussion on GB200 - batchwire - 10-07-2026
Quote:I'm trying to sanity-check my setup.
Interesting, I hadn't tried that. That matches what I saw on B200.
Re: Multimodal discussion on GB200 - fastlab - 10-08-2026
Ran a quick check on my box. The issue went away once I changed one setting and restarted the stack cleanly. I expected the model choice to matter more than it did. At this point the real bottleneck is my patience.
Re: Multimodal discussion on GB200 - dockbrook21 - 10-08-2026
What exact setup are you running? What model, what runtime, and roughly what context size?
Re: Multimodal discussion on GB200 - batchwire - 10-08-2026
following this because I'm in the same boat.
Re: Multimodal discussion on GB200 - rocmmixer - 10-08-2026
My first guess would be this: On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.
Re: Multimodal discussion on GB200 - nanogarden - 10-09-2026
A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?
Re: Multimodal discussion on GB200 - syshouse - 10-10-2026
I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.
Re: Multimodal discussion on GB200 - rocmmixer - 10-10-2026
Quote:I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.
I'd approach it in a boring order. If you're on Ollama, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: Multimodal discussion on GB200 - rocmriley - 10-10-2026
Not sure I agree there. I've seen people blame small local stacks when the real issue was much more mundane. Could be a workload difference, but I wouldn't generalize too fast from one setup.