+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Optimization & Performance (https://www.datasculpting.ai/inference/forumdisplay.php?fid=10)
+--- Forum: Context / KV Cache (https://www.datasculpting.ai/inference/forumdisplay.php?fid=41)
+--- Thread: speculative decoding actually worth it with LM Studio (/showthread.php?tid=908)
Pages:
1
2
speculative decoding actually worth it with LM Studio - cacheline - 10-08-2026
This keeps coming up in conversation so I figured I'd start a thread.
The more I work with local setups, the more it seems like KV cache pressure decides everything downstream.
Curious whether that lines up with what other people are seeing.
My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.
Re: speculative decoding actually worth it with LM Studio - rocmmixer - 10-10-2026
Good point. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: speculative decoding actually worth it with LM Studio - cacheline - 10-10-2026
What exact setup are you running? Are you testing this on GB200 or something else?
Re: speculative decoding actually worth it with LM Studio - bytelab - 10-12-2026
Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.
Re: speculative decoding actually worth it with LM Studio - fusedriver - 10-12-2026
Good point. I ended up in about the same place using SGLang.
Re: speculative decoding actually worth it with LM Studio - cacheline - 10-12-2026
Not sure I agree there. I get why that seems right, but on 5090 the bottleneck felt elsewhere. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.
Re: speculative decoding actually worth it with LM Studio - southtinker - 10-12-2026
I don't think that's the whole story. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.
Re: speculative decoding actually worth it with LM Studio - cacheline - 10-13-2026
Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Could be a workload difference, but I wouldn't generalize too fast from one setup. Turns out 'just one more test' was a lie.
Re: speculative decoding actually worth it with LM Studio - orbitgarden - 10-13-2026
I'd start with the simpler stack. Turns out 'just one more test' was a lie.
Re: speculative decoding actually worth it with LM Studio - mellowwire98 - 10-13-2026
I'd approach it in a boring order. On 4090, the easiest win is often to back off the aggressive settings and see whether the problem survives.