+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Drivers / Frameworks / Toolkits (https://www.datasculpting.ai/inference/forumdisplay.php?fid=8)
+--- Forum: Apple / MLX (https://www.datasculpting.ai/inference/forumdisplay.php?fid=33)
+--- Thread: Apple unified memory limits (/showthread.php?tid=253)
Pages:
1
2
Apple unified memory limits - thetaarch43 - 09-12-2026
Maybe I'm missing something obvious.
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling reranking.
Not looking for the perfect answer, mostly what worked in real use.
Re: Apple unified memory limits - batchsize - 09-12-2026
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at KV cache pressure. Not glamorous advice, but repeatable baselines save a lot of time.
I'm happy to test one more variant if you want a second data point.
Re: Apple unified memory limits - oxidestream - 09-13-2026
A couple things I'd check first. On A6000, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Re: Apple unified memory limits - mlxgarden - 09-13-2026
following this because I'm in the same boat. I keep promising myself this is the last rebuild.
Re: Apple unified memory limits - circuitwire58 - 09-13-2026
Maybe, but I had the opposite result. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Worth testing the simpler explanation before adopting the more interesting one.
Re: Apple unified memory limits - batchsize - 09-14-2026
What exact setup are you running? Is this local-only or are you routing anything externally?
Re: Apple unified memory limits - thetaarch43 - 09-15-2026
Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher.
Re: Apple unified memory limits - orbitline - 09-16-2026
worth checking the driver first.
Re: Apple unified memory limits - circuitwire58 - 09-16-2026
I hit almost the same issue last night.
Re: Apple unified memory limits - dockridge - 09-16-2026
subscribed, curious where this lands.