speculative decoding in llama.cpp

[cozykite]

[quote]Interesting, I hadn't tried that. That matches what I saw on 7900 XTX. I got better results once I stopped chasing one metric and just looked a

Maybe, but I had the opposite result. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Could be a workload difference, but I wouldn't generalize too fast from one setup.

If you post your exact settings, people here can probably narrow it down pretty fast.