+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Prompting & Context Engineering (https://www.datasculpting.ai/inference/forumdisplay.php?fid=15)
+--- Forum: Long Context / Memory (https://www.datasculpting.ai/inference/forumdisplay.php?fid=64)
+--- Thread: long context that stayed useful: what are you seeing? (/showthread.php?tid=1328)
long context that stayed useful: what are you seeing? - tensorriver - 10-17-2026
Spent some time on this today and wanted to compare notes.
I've gone back and forth between PyTorch and LM Studio for Llama. The interesting difference for me has been speculative decoding.
Interested in tradeoffs more than fan loyalty.
Re: long context that stayed useful: what are you seeing? - sanepeak48 - 10-17-2026
I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.
Re: long context that stayed useful: what are you seeing? - vrampilot - 10-18-2026
Not sure I agree there. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Worth testing the simpler explanation before adopting the more interesting one.
Re: long context that stayed useful: what are you seeing? - emberquill - 10-18-2026
following this because I'm in the same boat.
Re: long context that stayed useful: what are you seeing? - plainember52 - 10-19-2026
that fixed it for me too.
Re: long context that stayed useful: what are you seeing? - emberquill - 10-19-2026
That matches what I was seeing. That matches what I saw on 5090.
Re: long context that stayed useful: what are you seeing? - tinycontext - 10-19-2026
Quote:Not sure I agree there. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Worth testing the simpler expla
My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Phi and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.
Re: long context that stayed useful: what are you seeing? - latentscope35 - 10-19-2026
I think the bottleneck is somewhere else. I've seen people blame speculative decoding when the real issue was much more mundane. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.