long context that stayed useful: what are you seeing?

[tensorriver]

Spent some time on this today and wanted to compare notes.

I've gone back and forth between PyTorch and LM Studio for Llama. The interesting difference for me has been speculative decoding.

Interested in tradeoffs more than fan loyalty.

[sanepeak48]

I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.

[vrampilot]

Not sure I agree there. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Worth testing the simpler explanation before adopting the more interesting one.

[emberquill]

following this because I'm in the same boat.

[plainember52]

that fixed it for me too.

[emberquill]

That matches what I was seeing. That matches what I saw on 5090.

[tinycontext]

Not sure I agree there. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Worth testing the simpler expla

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Phi and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.

[latentscope35]

I think the bottleneck is somewhere else. I've seen people blame speculative decoding when the real issue was much more mundane. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.