prefix caching on repeated workflows

[cacheline]

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about workstation thermals.

Would rather learn from your scars than collect my own all at once.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

[cacheline]

That matches what I was seeing. That matches what I saw on M3 Max. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[kernelgrid]

subscribed, curious where this lands.

[cacheline]

subscribed, curious where this lands.

Thanks, that helps. The pattern around INT4 has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[cacheline]

subscribed, curious where this lands.

Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.

[oxidegrove40]

Thanks, that helps. That matches what I saw on 4090. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[cacheline]

Ran a quick check on my box. On dual 3090 with vLLM, I saw basically the same behavior. The main thing that moved was latency, not quality.

[sanepeak48]

Can you fill in a bit more context? Are you testing this on M3 Max or something else?

[cacheline]

Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.

[cacheline]

That matches what I was seeing. I ended up in about the same place using SGLang. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.