+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Optimization & Performance (https://www.datasculpting.ai/inference/forumdisplay.php?fid=10)
+--- Forum: Context / KV Cache (https://www.datasculpting.ai/inference/forumdisplay.php?fid=41)
+--- Thread: speculative decoding actually worth it with Ollama (/showthread.php?tid=1339)
speculative decoding actually worth it with Ollama - cacheline - 10-16-2026
Posting this in case it saves someone else a little time.
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around LM Studio and 5090.
Would rather learn from your scars than collect my own all at once.
Re: speculative decoding actually worth it with Ollama - cacheline - 10-16-2026
Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.
Re: speculative decoding actually worth it with Ollama - amberplane - 10-18-2026
worth checking the driver first.
Re: speculative decoding actually worth it with Ollama - cacheline - 10-18-2026
Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.
Re: speculative decoding actually worth it with Ollama - macropoint - 10-18-2026
What exact setup are you running? Is this local-only or are you routing anything externally?
Re: speculative decoding actually worth it with Ollama - phasescope - 10-18-2026
Quote:Interesting, I hadn't tried that. The pattern around prefix caching has been pretty similar for me. Worth rechecking your defaults though, because one
that fixed it for me too.
Re: speculative decoding actually worth it with Ollama - griffinloom66 - 10-19-2026
Interesting, I hadn't tried that. The pattern around prefix caching has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.