speculative decoding in llama.cpp

[northcraft90]

Need a little more detail here. Are you testing this on A6000 or something else? Turns out 'just one more test' was a lie.

[vrambandit]

A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?

[bytesupply76]

I think the bottleneck is somewhere else. I get why that seems right, but on A6000 the bottleneck felt elsewhere. Worth testing the simpler explanation before adopting the more interesting one.

[pixelcrate82]

That matches what I was seeing. The pattern around prefill/decode split has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.