speculative decoding actually worth it on 5090
|
speculative decoding actually worth it on 5090
|
09-24-2026, 10:35 AM
Quote:Did a bit of testing and thought I'd post the result here. Good point. The pattern around batch size has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer. Turns out 'just one more test' was a lie.
measure twice, benchmark three times
|
|
« Next Oldest | Next Newest »
|
| Messages In This Thread |
|
speculative decoding actually worth it on 5090 - by cacheline - 09-24-2026, 09:44 AM
Re: speculative decoding actually worth it on 5090 - by cacheline - 09-24-2026, 10:35 AM
Re: speculative decoding actually worth it on 5090 - by cacheline - 09-25-2026, 11:06 AM
Re: speculative decoding actually worth it on 5090 - by thetatinker77 - 09-25-2026, 12:44 PM
Re: speculative decoding actually worth it on 5090 - by cacheline - 09-27-2026, 01:04 AM
Re: speculative decoding actually worth it on 5090 - by cacheline - 09-28-2026, 01:27 PM
|
