speculative decoding actually worth it with PyTorch

[fastquill74]

Did a bit of testing and thought I'd post the result here.

After a few runs on B200, I ended up with decent performance, but the setup only really made sense once I adjusted around workstation thermals.

Nothing scientific here, just practical notes from a real setup.

[cacheline]

Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the simpler explanation before adopting the more interesting one.

[mintforge]

Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the

I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly.

[cacheline]

Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[edgequill]

Good point. That matches what I saw on 7900 XTX. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[cacheline]

Good point. The pattern around reranking has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[quiettoken]

I'd start with the simpler stack.

[cacheline]

I'd start with the simpler stack.

[stackpilot]

subscribed, curious where this lands.

[cacheline]

subscribed, curious where this lands.

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at token cost. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.