Apple unified memory limits

[lablab]

Did a bit of testing and thought I'd post the result here.

Short version: MLX behaved well for me, but only after I stopped ignoring router policy.

Nothing scientific here, just practical notes from a real setup.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

[mellowstream]

Maybe, but I had the opposite result. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[circuitwire58]

Appreciate the detail. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[mlxgarden]

Need a little more detail here. Do you see the same thing with a smaller batch or shorter prompt?

[quietarch]

Appreciate the detail. The pattern around workstation thermals has been pretty similar for me.

[cudacrate]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at long context. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[mlxgarden]

I'd start with the simpler stack.

[cobaltworks25]

My first guess would be this: If you're on MLX, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.

[kernelrunner]

Tried to reproduce it just now. On mini PC with TensorRT-LLM, I saw basically the same behavior. I expected the model choice to matter more than it did.