+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Prompting & Context Engineering (https://www.datasculpting.ai/inference/forumdisplay.php?fid=15)
+--- Forum: Long Context / Memory (https://www.datasculpting.ai/inference/forumdisplay.php?fid=64)
+--- Thread: long context that stayed useful (/showthread.php?tid=489)
Pages:
1
2
long context that stayed useful - ggufvault - 09-25-2026
I figured I'd ask here before changing everything.
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling tail latency.
Would appreciate practical advice rather than theory.
My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.
Re: long context that stayed useful - phasespark - 09-25-2026
Thanks, that helps. The pattern around speculative decoding has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: long context that stayed useful - ggufvault - 09-26-2026
I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Gemma and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.
Re: long context that stayed useful - amberkite - 09-29-2026
Appreciate the detail. That matches what I saw on M3 Max. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: long context that stayed useful - ggufvault - 09-29-2026
Good point. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: long context that stayed useful - edgequill - 09-30-2026
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tail latency. Once you know the plain path works, then add the faster path back in piece by piece.
Re: long context that stayed useful - flintriver50 - 10-01-2026
Quote:My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tail latency. Once you know t
Appreciate the detail. The pattern around quant drift has been pretty similar for me.
Re: long context that stayed useful - tinycontext - 10-01-2026
Quote:[quote]My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tail latency. Once you
Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.
Re: long context that stayed useful - flintriver50 - 10-02-2026
Quote:[quote][quote]My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tail latency. O
I'd approach it in a boring order. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Re: long context that stayed useful - quiethouse - 10-02-2026
A couple questions before I guess too hard. What model, what runtime, and roughly what context size?