Tensor cores actually helping here

[amberpatch25]

I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the simpler explanation before adopting the more interesting one.