speculative decoding in llama.cpp

[mellowwire98]

I don't think that's the whole story. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.

[vectorport]

Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?

[torchdesk]

Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?

That matches what I was seeing. That matches what I saw on mini PC. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[bytesupply76]

I hit almost the same issue last night.