throughput per watt reality check for DeepSeek

[cacheline]

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: PyTorch on GB200, keeping batch size in mind from the start.

Maybe useful as a starting point for newer folks.

[turboscope5]

Thought I'd post a small recipe because a lot of threads ask similar things.

That matches what I was seeing. That matches what I saw on A6000. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[edgeriver]

Maybe, but I had the opposite result. I get why that seems right, but on B200 the bottleneck felt elsewhere. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[northcraft90]

Appreciate the detail. I ended up in about the same place using LM Studio. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[cachebench]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at small local stacks. Not glamorous advice, but repeatable baselines save a lot of time.

[cacheworks]

worth checking the driver first.

[marbletrack]

Maybe, but I had the opposite result. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[kernelpeak]

Not sure I agree there. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.

[circuitworks90]

Not sure I agree there. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.

Need a little more detail here. Do you see the same thing with a smaller batch or shorter prompt?