SBC inference experiments on GB200

[mintquill]

Posting this in case it saves someone else a little time.

I'm not trying to build a perfect architecture diagram. I just want a sane starting point around PyTorch and Threadripper box.

Simple and boring is a valid answer.

This is mostly for local work rather than public serving.

[homelabhenry]

worth checking the driver first.

[thetatinker77]

Thanks, that helps. That matches what I saw on dual 3090.

[cobalttinker]

That matches what I was seeing. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[vramghost]

That matches what I was seeing. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at wh

My first guess would be this: On dual 3090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.

[lablab]

A couple things I'd check first. On M3 Max, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

[homelabhenry]

Not sure I agree there. I've seen people blame consumer GPUs when the real issue was much more mundane. Worth testing the simpler explanation before adopting the more interesting one.

[thetatinker77]

Appreciate the detail. The pattern around token cost has been pretty similar for me.

[homelabhenry]

that fixed it for me too.