+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: llama.cpp / GGUF (https://www.datasculpting.ai/inference/forumdisplay.php?fid=35)
+--- Thread: speculative decoding in llama.cpp: anyone testing this? (/showthread.php?tid=534)
Pages:
1
2
speculative decoding in llama.cpp: anyone testing this? - mellowwire98 - 09-27-2026
Spent some time on this today and wanted to compare notes.
I've gone back and forth between MLX and MLX for Gemma. The interesting difference for me has been prefill/decode split.
Trying to avoid rebuilding the whole thing twice.
Re: speculative decoding in llama.cpp: anyone testing this? - pixelcrate82 - 09-28-2026
Appreciate the detail. I ended up in about the same place using vLLM. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: speculative decoding in llama.cpp: anyone testing this? - macroforge - 09-28-2026
Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.
Re: speculative decoding in llama.cpp: anyone testing this? - mangotinker73 - 09-28-2026
A couple questions before I guess too hard. Are you testing this on mini PC or something else?
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.
Re: speculative decoding in llama.cpp: anyone testing this? - kernelpeak - 09-29-2026
That matches what I was seeing. That matches what I saw on GB200.
Re: speculative decoding in llama.cpp: anyone testing this? - bytesupply76 - 09-29-2026
Quote:That matches what I was seeing. That matches what I saw on GB200.
worth checking the driver first.
Re: speculative decoding in llama.cpp: anyone testing this? - mangotinker73 - 10-02-2026
Quote:Quote:That matches what I was seeing. That matches what I saw on GB200.
A couple things I'd check first. On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.
Re: speculative decoding in llama.cpp: anyone testing this? - macroforge - 10-02-2026
Interesting, I hadn't tried that. The pattern around router policy has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: speculative decoding in llama.cpp: anyone testing this? - visionyard23 - 10-02-2026
Quote:Interesting, I hadn't tried that. The pattern around router policy has been pretty similar for me. The biggest change for me was simplifying the stack
Not sure I agree there. I've seen people blame INT4 when the real issue was much more mundane. Could be a workload difference, but I wouldn't generalize too fast from one setup.
I'm happy to test one more variant if you want a second data point. At this point the real bottleneck is my patience.
Re: speculative decoding in llama.cpp: anyone testing this? - vectorport - 10-04-2026
Interesting, I hadn't tried that. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.