+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: llama.cpp / GGUF (https://www.datasculpting.ai/inference/forumdisplay.php?fid=35)
+--- Thread: speculative decoding in llama.cpp (/showthread.php?tid=283)
Pages:
1
2
Re: speculative decoding in llama.cpp - northcraft90 - 09-24-2026
Need a little more detail here. Are you testing this on A6000 or something else? Turns out 'just one more test' was a lie.
Re: speculative decoding in llama.cpp - vrambandit - 09-24-2026
A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?
Re: speculative decoding in llama.cpp - bytesupply76 - 09-26-2026
I think the bottleneck is somewhere else. I get why that seems right, but on A6000 the bottleneck felt elsewhere. Worth testing the simpler explanation before adopting the more interesting one.
Re: speculative decoding in llama.cpp - pixelcrate82 - 09-26-2026
That matches what I was seeing. The pattern around prefill/decode split has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.