speculative decoding in llama.cpp

[dockridge]

[quote]My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Mixtral and compare from there. A lot of these threa

Not sure I agree there. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. I keep promising myself this is the last rebuild.