Tensor cores actually helping here

[thetapath]

worth checking the driver first.

[edgecoast]

Appreciate the detail. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[dockember]

I'd approach it in a boring order. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious.

[orbitbench6]

Posting this in case it saves someone else a little time.

If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

I'm especially interested in mistakes worth avoiding.

I care more about stability than squeezing the last possible tokens/sec out of it.

[amberpatch25]

Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?

[edgecoast]

Ran a quick check on my box. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.

[thetapath]

My first guess would be this: On GB200, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[amberpatch25]

I hit almost the same issue last night.