Multimodal discussion on GB200

[southtinker]

I'm trying to sanity-check my setup.

I'm mostly working with Mistral in MLX on 5090. The part I'm trying to understand is continuous batching.

Not looking for the perfect answer, mostly what worked in real use.

I care more about stability than squeezing the last possible tokens/sec out of it.

[batchwire]

I'm trying to sanity-check my setup.

Interesting, I hadn't tried that. That matches what I saw on B200.

[fastlab]

Ran a quick check on my box. The issue went away once I changed one setting and restarted the stack cleanly. I expected the model choice to matter more than it did. At this point the real bottleneck is my patience.

[dockbrook21]

What exact setup are you running? What model, what runtime, and roughly what context size?

[batchwire]

following this because I'm in the same boat.

[rocmmixer]

My first guess would be this: On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.

[nanogarden]

A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?

[syshouse]

I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.

[rocmmixer]

I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.

I'd approach it in a boring order. If you're on Ollama, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[rocmriley]

Not sure I agree there. I've seen people blame small local stacks when the real issue was much more mundane. Could be a workload difference, but I wouldn't generalize too fast from one setup.