+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Optimization & Performance (https://www.datasculpting.ai/inference/forumdisplay.php?fid=10)
+--- Forum: Context / KV Cache (https://www.datasculpting.ai/inference/forumdisplay.php?fid=41)
+--- Thread: speculative decoding actually worth it with PyTorch (/showthread.php?tid=64)
Pages:
1
2
Re: speculative decoding actually worth it with PyTorch - mintforge - 09-01-2026
My first guess would be this: On M3 Max, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Re: speculative decoding actually worth it with PyTorch - edgecoast - 09-01-2026
Quote:My first guess would be this: On M3 Max, the easiest win is often to back off the aggressive settings and see whether the problem survives.
I don't think that's the whole story. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Could be a workload difference, but I wouldn't generalize too fast from one setup.
I'm happy to test one more variant if you want a second data point.