speculative decoding in llama.cpp

[torchdesk]

Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?

That matches what I was seeing. That matches what I saw on mini PC. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.