Blackwell deployment chatter

[edgecoast]

I figured I'd ask here before changing everything.

I'm mostly working with Mistral in Ollama on 4090. The part I'm trying to understand is prefill/decode split.

If you've tested something similar, what did you keep and what did you stop bothering with?

[sparrowkite53]

I figured I'd ask here before changing everything.

Maybe, but I had the opposite result. I've seen people blame MoE serving when the real issue was much more mundane.

[kernelmike]

Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?

[amberpatch25]

I tested this tonight. On B200 with TensorRT-LLM, I saw basically the same behavior. The main thing that moved was latency, not quality.

[rocmmixer]

I think the bottleneck is somewhere else. I've seen people blame NVFP4 when the real issue was much more mundane. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[kernelmike]

I think the bottleneck is somewhere else. I've seen people blame NVFP4 when the real issue was much more mundane. Could be a workload difference, but

What worked for me was pretty simple. If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.

[latentgarden]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Phi and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.

[dockember]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Gemma and compare from there.

[labline]

What exact setup are you running? Are you testing this on GB200 or something else?

[amberpatch25]

that fixed it for me too.