what truly stays local on B200

[mlxgarden]

I think the bottleneck is somewhere else. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.

[modgarden]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at consumer GPUs. Not glamorous advice, but repeatable baselines save a lot of time.

[mintstation99]

worth checking the driver first.

If you post your exact settings, people here can probably narrow it down pretty fast.

[vectorport]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at reranking. Once you know the plain path works, then add the faster path back in piece by piece.