disaggregated inference thoughts on M3 Max

[cozyarch]

Has anyone else run into this?

My current setup is SGLang with 4090, and I keep running into prefix caching.

If you've tested something similar, what did you keep and what did you stop bothering with?

[edgeriver]

Appreciate the detail. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer. Turns out 'just one more test' was a lie.

[tensorfield88]

That matches what I was seeing. That matches what I saw on 5090.

[labline]

Tried to reproduce it just now. On A6000 with LM Studio, I saw basically the same behavior. I expected the model choice to matter more than it did.

[kernelmike]

Maybe, but I had the opposite result. I get why that seems right, but on M3 Max the bottleneck felt elsewhere. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[cudapeak]

Tried to reproduce it just now. On GB200 with Ollama, I saw basically the same behavior.

[cozygrid12]

that fixed it for me too.

[kernelmike]

that fixed it for me too.

Interesting, I hadn't tried that. The pattern around MoE serving has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[rocmhouse90]

Appreciate the detail. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

If you post your exact settings, people here can probably narrow it down pretty fast.

[cudapeak]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Qwen and compare from there.