+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: llama.cpp / GGUF (https://www.datasculpting.ai/inference/forumdisplay.php?fid=35)
+--- Thread: speculative decoding in llama.cpp (/showthread.php?tid=1424)
Pages:
1
2
speculative decoding in llama.cpp - bytesupply76 - 10-17-2026
This keeps coming up in conversation so I figured I'd start a thread.
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with KV cache pressure.
Curious whether that lines up with what other people are seeing.
Re: speculative decoding in llama.cpp - embershed99 - 10-18-2026
same here on my side.
Re: speculative decoding in llama.cpp - vectorport - 10-18-2026
Quote:same here on my side.
That matches what I was seeing. That matches what I saw on GB200. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: speculative decoding in llama.cpp - mangotinker73 - 10-18-2026
Did a small comparison this afternoon. On M3 Max with MLX, I saw basically the same behavior. The main thing that moved was latency, not quality.
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.
Re: speculative decoding in llama.cpp - embershed99 - 10-18-2026
I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.
Re: speculative decoding in llama.cpp - mintforge - 10-18-2026
Ran a quick check on my box. On GB200 with SGLang, I saw basically the same behavior.
Re: speculative decoding in llama.cpp - thetagrid15 - 10-18-2026
A couple questions before I guess too hard. What model, what runtime, and roughly what context size?
Re: speculative decoding in llama.cpp - pixelcrate82 - 10-18-2026
I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at structured output. Not glamorous advice, but repeatable baselines save a lot of time.
Re: speculative decoding in llama.cpp - pixelcrate82 - 10-18-2026
Quote:Quote:same here on my side.
I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Llama and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.
Re: speculative decoding in llama.cpp - oxidestream - 10-19-2026
I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly.