speculative decoding actually worth it with PyTorch

[mintforge]

My first guess would be this: On M3 Max, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[edgecoast]

My first guess would be this: On M3 Max, the easiest win is often to back off the aggressive settings and see whether the problem survives.

I don't think that's the whole story. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Could be a workload difference, but I wouldn't generalize too fast from one setup.

I'm happy to test one more variant if you want a second data point.