+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Drivers / Frameworks / Toolkits (https://www.datasculpting.ai/inference/forumdisplay.php?fid=8)
+--- Forum: Apple / MLX (https://www.datasculpting.ai/inference/forumdisplay.php?fid=33)
+--- Thread: Apple unified memory limits (/showthread.php?tid=1262)
Pages:
1
2
Apple unified memory limits - mlxgarden - 10-16-2026
Did a bit of testing and thought I'd post the result here.
Short version: TensorRT-LLM behaved well for me, but only after I stopped ignoring HBM constraints.
Curious if anyone else landed in the same place.
Re: Apple unified memory limits - orbitline - 10-16-2026
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at continuous batching. Not glamorous advice, but repeatable baselines save a lot of time.
Re: Apple unified memory limits - cudaworks - 10-16-2026
subscribed, curious where this lands.
Re: Apple unified memory limits - mlxgarden - 10-17-2026
A couple things I'd check first. If you're on MLX, I'd confirm the version first and then reduce variables until the behavior is obvious.
Re: Apple unified memory limits - thetaarch43 - 10-17-2026
What worked for me was pretty simple. On B200, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Re: Apple unified memory limits - quietarch - 10-17-2026
following this because I'm in the same boat.
Re: Apple unified memory limits - dockridge - 10-18-2026
I tested this tonight. On 7900 XTX with vLLM, I saw basically the same behavior. Turns out 'just one more test' was a lie.
Re: Apple unified memory limits - circuitwire58 - 10-18-2026
I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at MoE serving. Not glamorous advice, but repeatable baselines save a lot of time.
Re: Apple unified memory limits - cudacrate - 10-18-2026
Quote:I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at MoE serving. Not glamoro
I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. I expected the model choice to matter more than it did.
Re: Apple unified memory limits - turboyard94 - 10-18-2026
I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.