+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Optimization & Performance (https://www.datasculpting.ai/inference/forumdisplay.php?fid=10)
+--- Forum: Context / KV Cache (https://www.datasculpting.ai/inference/forumdisplay.php?fid=41)
+--- Thread: paged attention observations (/showthread.php?tid=727)
paged attention observations - cacheline - 10-04-2026
This keeps coming up in conversation so I figured I'd start a thread.
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with workstation thermals.
Maybe obvious to some of you, but it clicked for me this week.
My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.
Re: paged attention observations - quantyard - 10-04-2026
that fixed it for me too.
Re: paged attention observations - cacheline - 10-04-2026
Quote:that fixed it for me too.
Thanks, that helps. The pattern around NVFP4 has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: paged attention observations - vectorcrate98 - 10-04-2026
Quote:Quote:that fixed it for me too.
What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefix caching.
Re: paged attention observations - sparrowghost - 10-05-2026
A couple things I'd check first. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: paged attention observations - cacheline - 10-05-2026
Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.
Re: paged attention observations - fusedriver - 10-06-2026
Quote:Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up
Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal.
Re: paged attention observations - cacheline - 10-06-2026
A couple questions before I guess too hard. What model, what runtime, and roughly what context size?