+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Optimization & Performance (https://www.datasculpting.ai/inference/forumdisplay.php?fid=10)
+--- Forum: Context / KV Cache (https://www.datasculpting.ai/inference/forumdisplay.php?fid=41)
+--- Thread: speculative decoding actually worth it (/showthread.php?tid=158)
Pages:
1
2
speculative decoding actually worth it - cacheline - 09-06-2026
I'm trying to sanity-check my setup.
My current setup is TensorRT-LLM with 4090, and I keep running into continuous batching.
If you've tested something similar, what did you keep and what did you stop bothering with?
This is mostly for local work rather than public serving.
Re: speculative decoding actually worth it - cacheline - 09-06-2026
Quote:I'm trying to sanity-check my setup.
Not sure I agree there. I get why that seems right, but on 4090 the bottleneck felt elsewhere. Worth testing the simpler explanation before adopting the more interesting one.
Re: speculative decoding actually worth it - torchsmith25 - 09-09-2026
Quote:Quote:I'm trying to sanity-check my setup.
What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at HBM constraints. At this point the real bottleneck is my patience.
Re: speculative decoding actually worth it - cacheline - 09-09-2026
Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.
Re: speculative decoding actually worth it - vectorember - 09-09-2026
Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal.
Re: speculative decoding actually worth it - cacheline - 09-09-2026
Quote:Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal.
Good point. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: speculative decoding actually worth it - amberpatch25 - 09-09-2026
Need a little more detail here. Is this local-only or are you routing anything externally?
Re: speculative decoding actually worth it - cacheline - 09-09-2026
My first guess would be this: If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: speculative decoding actually worth it - visionforge - 09-09-2026
Quote:My first guess would be this: If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of th
worth checking the driver first.
Re: speculative decoding actually worth it - cacheline - 09-09-2026
Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.