speculative decoding actually worth it with PyTorch
|
speculative decoding actually worth it with PyTorch
|
10-08-2026, 06:18 AM
Quote:Maybe, but I had the opposite result. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Could be a worklo A couple questions before I guess too hard. What model, what runtime, and roughly what context size?
measure twice, benchmark three times
|
|
« Next Oldest | Next Newest »
|
| Messages In This Thread |
|
speculative decoding actually worth it with PyTorch - by cacheline - 10-08-2026, 05:48 AM
Re: speculative decoding actually worth it with PyTorch - by cacheline - 10-08-2026, 06:09 AM
Re: speculative decoding actually worth it with PyTorch - by cacheline - 10-08-2026, 06:18 AM
Re: speculative decoding actually worth it with PyTorch - by quantyard - 10-08-2026, 07:27 AM
Re: speculative decoding actually worth it with PyTorch - by cacheline - 10-08-2026, 08:42 AM
Re: speculative decoding actually worth it with PyTorch - by vrambandit - 10-08-2026, 05:26 PM
Re: speculative decoding actually worth it with PyTorch - by cacheline - 10-11-2026, 05:26 PM
|
