PyTorch compile got faster? on dual 3090

[phaseport36]

subscribed, curious where this lands.

I'm happy to test one more variant if you want a second data point.

[byteyard]

Spent some time on this today and wanted to compare notes.

Two paths both seem viable, but they break down differently once structured output becomes important.

Trying to avoid rebuilding the whole thing twice.

[vrambandit]

A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?

[circuitwire58]

A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?

[batchscope72]

Thanks, that helps. I ended up in about the same place using LM Studio.

[dockember]

Not sure I agree there. I've seen people blame tokens/sec when the real issue was much more mundane.

[mintstream9]

That matches what I was seeing. The pattern around quant drift has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.