disaggregated inference thoughts on M3 Max

[kernelmike]

subscribed, curious where this lands.

[quantsupply]

Interesting, I hadn't tried that. I don't think your conclusion is far off.

[macropilot]

Can you fill in a bit more context? What model, what runtime, and roughly what context size?

[kernelmike]

Maybe, but I had the opposite result. I've seen people blame consumer GPUs when the real issue was much more mundane. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[tensorfield88]

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Llama and compare from there.