prefill/decode separation in practice with PyTorch

[thetashed24]

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on M3 Max, or squeeze more out of the stack and deal with the complexity.

Interested in tradeoffs more than fan loyalty.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

[cacheline]

I don't think that's the whole story. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the simpler explanation before adopting the more interesting one.

[routearch]

I don't think that's the whole story. I've seen people blame VRAM headroom when the real issue was much more mundane. Worth testing the simpler explanation before adopting the more interesting one. Turns out 'just one more test' was a lie.