multi-GPU offload advice

[vectorport]

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on GB200, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?

I care more about stability than squeezing the last possible tokens/sec out of it.

[signalpeak]

Appreciate the detail. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer. Turns out 'just one more test' was a lie.

[northcraft90]

Thanks, that helps. The pattern around prefill/decode split has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.