+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Drivers / Frameworks / Toolkits (https://www.datasculpting.ai/inference/forumdisplay.php?fid=8)
+--- Forum: NVIDIA / CUDA (https://www.datasculpting.ai/inference/forumdisplay.php?fid=31)
+--- Thread: Blackwell deployment chatter (/showthread.php?tid=1231)
Blackwell deployment chatter - edgecoast - 10-14-2026
I figured I'd ask here before changing everything.
I'm mostly working with Mistral in Ollama on 4090. The part I'm trying to understand is prefill/decode split.
If you've tested something similar, what did you keep and what did you stop bothering with?
Re: Blackwell deployment chatter - sparrowkite53 - 10-16-2026
Quote:I figured I'd ask here before changing everything.
Maybe, but I had the opposite result. I've seen people blame MoE serving when the real issue was much more mundane.
Re: Blackwell deployment chatter - kernelmike - 10-16-2026
Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?
Re: Blackwell deployment chatter - amberpatch25 - 10-17-2026
I tested this tonight. On B200 with TensorRT-LLM, I saw basically the same behavior. The main thing that moved was latency, not quality.
Re: Blackwell deployment chatter - rocmmixer - 10-18-2026
I think the bottleneck is somewhere else. I've seen people blame NVFP4 when the real issue was much more mundane. Could be a workload difference, but I wouldn't generalize too fast from one setup.
Re: Blackwell deployment chatter - kernelmike - 10-18-2026
Quote:I think the bottleneck is somewhere else. I've seen people blame NVFP4 when the real issue was much more mundane. Could be a workload difference, but
What worked for me was pretty simple. If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.
Re: Blackwell deployment chatter - latentgarden - 10-18-2026
A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Phi and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.
Re: Blackwell deployment chatter - dockember - 10-18-2026
A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Gemma and compare from there.
Re: Blackwell deployment chatter - labline - 10-18-2026
What exact setup are you running? Are you testing this on GB200 or something else?
Re: Blackwell deployment chatter - amberpatch25 - 10-19-2026
that fixed it for me too.