speculative decoding in llama.cpp

[mellowwire98]

Posting this in case it saves someone else a little time.

If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Simple and boring is a valid answer.

[phasescope]

I'd start with the simpler stack.

[vrambandit]

Interesting, I hadn't tried that. The pattern around INT4 has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[mellowwire98]

worth checking the driver first.

[vrambandit]

same here on my side.

[batchfield55]

subscribed, curious where this lands.

I'm happy to test one more variant if you want a second data point.

[pixelcrate82]

What exact setup are you running? Do you see the same thing with a smaller batch or shorter prompt?

[vrambandit]

Did a small comparison this afternoon. On M3 Max with Ollama, I saw basically the same behavior. The main thing that moved was latency, not quality. At this point the real bottleneck is my patience.

[mellowwire98]

Interesting, I hadn't tried that. The pattern around token cost has been pretty similar for me.