PyTorch compile got faster?

[routecraft]

Appreciate the detail. The pattern around long context has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[plainscope50]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

I'm happy to test one more variant if you want a second data point.

[mintstation99]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at continuous batching. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[routecrate]

A couple things I'd check first. On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives.