Inference › Optimization & Performance › Context / KV Cache v
speculative decoding actually worth it with Ollama

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Linear Mode
speculative decoding actually worth it with Ollama
cacheline Offline
Senior Member
****
Posts: 341
Threads: 51
Joined: Sep 2026
Reputation: 0
#2
10-16-2026, 12:14 PM
Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.
measure twice, benchmark three times
Find
Reply


Messages In This Thread
Re: speculative decoding actually worth it with Ollama - by cacheline - 10-16-2026, 12:14 PM

Forum Jump:


Users browsing this thread: 1 Guest(s)