+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Optimization & Performance (https://www.datasculpting.ai/inference/forumdisplay.php?fid=10)
+--- Forum: Context / KV Cache (https://www.datasculpting.ai/inference/forumdisplay.php?fid=41)
+--- Thread: prefix caching on repeated workflows (/showthread.php?tid=1076)
Pages:
1
2
prefix caching on repeated workflows - cacheline - 10-11-2026
Posting this in case it saves someone else a little time.
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about workstation thermals.
Would rather learn from your scars than collect my own all at once.
My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.
Re: prefix caching on repeated workflows - cacheline - 10-11-2026
That matches what I was seeing. That matches what I saw on M3 Max. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: prefix caching on repeated workflows - kernelgrid - 10-14-2026
subscribed, curious where this lands.
Re: prefix caching on repeated workflows - cacheline - 10-17-2026
Quote:subscribed, curious where this lands.
Thanks, that helps. The pattern around INT4 has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: prefix caching on repeated workflows - cacheline - 10-17-2026
Quote:Quote:subscribed, curious where this lands.
Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.
Re: prefix caching on repeated workflows - oxidegrove40 - 10-17-2026
Thanks, that helps. That matches what I saw on 4090. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: prefix caching on repeated workflows - cacheline - 10-18-2026
Ran a quick check on my box. On dual 3090 with vLLM, I saw basically the same behavior. The main thing that moved was latency, not quality.
Re: prefix caching on repeated workflows - sanepeak48 - 10-18-2026
Can you fill in a bit more context? Are you testing this on M3 Max or something else?
Re: prefix caching on repeated workflows - cacheline - 10-19-2026
Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.
Re: prefix caching on repeated workflows - cacheline - 10-19-2026
That matches what I was seeing. I ended up in about the same place using SGLang. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.