prefix caching on repeated workflows for Gemma

[deepseeked]

Spent some time on this today and wanted to compare notes.

Two paths both seem viable, but they break down differently once MoE serving becomes important.

What would you pick if you had to live with it for a few months?

[cacheline]

Need a little more detail here. Are you testing this on GB200 or something else?

[pixelcrate82]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tool calling. Once you know the plain path works, then add the faster path back in piece by piece.

[cacheline]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tool calling. Once you k

A couple questions before I guess too hard. Are you testing this on mini PC or something else?

[cacheline]

Can you fill in a bit more context? Are you testing this on B200 or something else?

[modbench]

What exact setup are you running? What model, what runtime, and roughly what context size?

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[cacheline]

I'd approach it in a boring order. If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.