+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: llama.cpp / GGUF (https://www.datasculpting.ai/inference/forumdisplay.php?fid=35)
+--- Thread: speculative decoding in llama.cpp (/showthread.php?tid=271)
speculative decoding in llama.cpp - mellowwire98 - 09-13-2026
Posting this in case it saves someone else a little time.
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?
Simple and boring is a valid answer.
Re: speculative decoding in llama.cpp - phasescope - 09-14-2026
I'd start with the simpler stack.
Re: speculative decoding in llama.cpp - vrambandit - 09-14-2026
Interesting, I hadn't tried that. The pattern around INT4 has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: speculative decoding in llama.cpp - mellowwire98 - 09-14-2026
worth checking the driver first.
Re: speculative decoding in llama.cpp - vrambandit - 09-17-2026
same here on my side.
Re: speculative decoding in llama.cpp - batchfield55 - 09-18-2026
subscribed, curious where this lands.
I'm happy to test one more variant if you want a second data point.
Re: speculative decoding in llama.cpp - pixelcrate82 - 09-18-2026
What exact setup are you running? Do you see the same thing with a smaller batch or shorter prompt?
Re: speculative decoding in llama.cpp - vrambandit - 09-20-2026
Did a small comparison this afternoon. On M3 Max with Ollama, I saw basically the same behavior. The main thing that moved was latency, not quality. At this point the real bottleneck is my patience.
Re: speculative decoding in llama.cpp - mellowwire98 - 09-20-2026
Interesting, I hadn't tried that. The pattern around token cost has been pretty similar for me.