+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: llama.cpp / GGUF (https://www.datasculpting.ai/inference/forumdisplay.php?fid=35)
+--- Thread: speculative decoding in llama.cpp (/showthread.php?tid=693)
Pages:
1
2
speculative decoding in llama.cpp - dockridge - 10-03-2026
Wanted to compare notes with people doing similar things.
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling MoE serving.
Not looking for the perfect answer, mostly what worked in real use.
Re: speculative decoding in llama.cpp - visionyard23 - 10-03-2026
same here on my side.
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.
Re: speculative decoding in llama.cpp - macroforge - 10-03-2026
Appreciate the detail. I ended up in about the same place using PyTorch. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: speculative decoding in llama.cpp - vrambandit - 10-06-2026
Ran a quick check on my box. On 5090 with llama.cpp, I saw basically the same behavior. I expected the model choice to matter more than it did. At this point the real bottleneck is my patience.
Re: speculative decoding in llama.cpp - phasescope - 10-06-2026
My first guess would be this: On B200, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Re: speculative decoding in llama.cpp - northcraft90 - 10-06-2026
same here on my side.
Re: speculative decoding in llama.cpp - dockridge - 10-06-2026
What worked for me was pretty simple. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious.
Re: speculative decoding in llama.cpp - vrambandit - 10-07-2026
Need a little more detail here. What model, what runtime, and roughly what context size?
Re: speculative decoding in llama.cpp - phasescope - 10-07-2026
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at router policy.
Re: speculative decoding in llama.cpp - latentcraft - 10-07-2026
Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.