throughput per watt reality check for Qwen

[kernelmike]

What exact setup are you running? Do you see the same thing with a smaller batch or shorter prompt?

[stackarch]

Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.

[lablab]

I hit almost the same issue last night.

[vectorhouse71]

I hit almost the same issue last night.

I don't think that's the whole story. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[kernelmike]

I hit almost the same issue last night.

Appreciate the detail. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[circuitworks90]

I hit almost the same issue last night.

That matches what I was seeing. That matches what I saw on B200. The biggest change for me was simplifying the stack instead of adding one more optimization layer.