Apple unified memory limits

[mlxgarden]

Did a bit of testing and thought I'd post the result here.

Short version: TensorRT-LLM behaved well for me, but only after I stopped ignoring HBM constraints.

Curious if anyone else landed in the same place.

[orbitline]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at continuous batching. Not glamorous advice, but repeatable baselines save a lot of time.

[cudaworks]

subscribed, curious where this lands.

[mlxgarden]

A couple things I'd check first. If you're on MLX, I'd confirm the version first and then reduce variables until the behavior is obvious.

[thetaarch43]

What worked for me was pretty simple. On B200, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[quietarch]

following this because I'm in the same boat.

[dockridge]

I tested this tonight. On 7900 XTX with vLLM, I saw basically the same behavior. Turns out 'just one more test' was a lie.

[circuitwire58]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at MoE serving. Not glamorous advice, but repeatable baselines save a lot of time.

[cudacrate]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at MoE serving. Not glamoro

I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. I expected the model choice to matter more than it did.

[turboyard94]

I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.