+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Prompting & Context Engineering (https://www.datasculpting.ai/inference/forumdisplay.php?fid=15)
+--- Forum: Long Context / Memory (https://www.datasculpting.ai/inference/forumdisplay.php?fid=64)
+--- Thread: memory layers vs plain long context: setup advice (/showthread.php?tid=571)
Pages:
1
2
memory layers vs plain long context: setup advice - plainember52 - 09-28-2026
This keeps coming up in conversation so I figured I'd start a thread.
The more I work with local setups, the more it seems like speculative decoding decides everything downstream.
Wondering if this is just where the whole space is heading now.
Re: memory layers vs plain long context: setup advice - mellowsupply - 09-28-2026
Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal.
Re: memory layers vs plain long context: setup advice - tinycontext - 09-28-2026
Interesting, I hadn't tried that. The pattern around tool calling has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: memory layers vs plain long context: setup advice - torchkite - 09-29-2026
Quote:Interesting, I hadn't tried that. The pattern around tool calling has been pretty similar for me. Worth rechecking your defaults though, because one b
Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. I expected the model choice to matter more than it did.
Re: memory layers vs plain long context: setup advice - mellowsupply - 09-30-2026
subscribed, curious where this lands.
Re: memory layers vs plain long context: setup advice - quiethouse - 09-30-2026
Thanks, that helps. The pattern around KV cache pressure has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: memory layers vs plain long context: setup advice - turboscope5 - 09-30-2026
Did a small comparison this afternoon. On 7900 XTX with Ollama, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.
Re: memory layers vs plain long context: setup advice - fusedhall56 - 09-30-2026
What exact setup are you running? Is this local-only or are you routing anything externally?
Re: memory layers vs plain long context: setup advice - tinycontext - 10-03-2026
Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.
Re: memory layers vs plain long context: setup advice - phasescope - 10-04-2026
Quote:Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure
subscribed, curious where this lands.