PyTorch compile got faster?: worth it?

[kernelmike]

Spent some time on this today and wanted to compare notes.

Two paths both seem viable, but they break down differently once batch size becomes important.

Interested in tradeoffs more than fan loyalty.

This is mostly for local work rather than public serving.

[vectorfield]

Spent some time on this today and wanted to compare notes.

I'd approach it in a boring order. On 7900 XTX, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

[quietstack]

What exact setup are you running? Is this local-only or are you routing anything externally?

[paperrunner]

worth checking the driver first.

[mlxgarden]

Good point. I don't think your conclusion is far off.

[amberpatch25]

Need a little more detail here. Is this local-only or are you routing anything externally?

[kernelghost]

Good point. I ended up in about the same place using TensorRT-LLM. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[torchkite]

Thanks, that helps. I ended up in about the same place using vLLM. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[localtinker]

Thanks, that helps. I ended up in about the same place using vLLM. I got better results once I stopped chasing one metric and just looked at whether t

Appreciate the detail. I ended up in about the same place using vLLM. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[macrostack69]

My first guess would be this: On GB200, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.