+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Optimization & Performance (https://www.datasculpting.ai/inference/forumdisplay.php?fid=10)
+--- Forum: Context / KV Cache (https://www.datasculpting.ai/inference/forumdisplay.php?fid=41)
+--- Thread: prefix caching on repeated workflows for Gemma (/showthread.php?tid=1181)
prefix caching on repeated workflows for Gemma - deepseeked - 10-13-2026
Spent some time on this today and wanted to compare notes.
Two paths both seem viable, but they break down differently once MoE serving becomes important.
What would you pick if you had to live with it for a few months?
Re: prefix caching on repeated workflows for Gemma - cacheline - 10-13-2026
Need a little more detail here. Are you testing this on GB200 or something else?
Re: prefix caching on repeated workflows for Gemma - pixelcrate82 - 10-13-2026
I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tool calling. Once you know the plain path works, then add the faster path back in piece by piece.
Re: prefix caching on repeated workflows for Gemma - cacheline - 10-13-2026
Quote:I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tool calling. Once you k
A couple questions before I guess too hard. Are you testing this on mini PC or something else?
Re: prefix caching on repeated workflows for Gemma - cacheline - 10-14-2026
Can you fill in a bit more context? Are you testing this on B200 or something else?
Re: prefix caching on repeated workflows for Gemma - modbench - 10-15-2026
What exact setup are you running? What model, what runtime, and roughly what context size?
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.
Re: prefix caching on repeated workflows for Gemma - cacheline - 10-15-2026
I'd approach it in a boring order. If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.