Apple unified memory limits with vLLM

[mlxgarden]

Maybe I'm missing something obvious.

I'm mostly working with Phi in llama.cpp on GB200. The part I'm trying to understand is NVFP4.

Would appreciate practical advice rather than theory.

[dockridge]

Maybe I'm missing something obvious.

worth checking the driver first.

[mlxgarden]

Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher.

[plainember52]

Maybe, but I had the opposite result. I've seen people blame consumer GPUs when the real issue was much more mundane. Worth testing the simpler explanation before adopting the more interesting one.

[dockridge]

I tested this tonight. My result was mixed: first run looked bad, second run looked normal.

[thetaarch43]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tool calling.

[torchdesk]

Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.

[batchsize]

that fixed it for me too.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[cobaltlab]

Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal.

[cudacrate]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at VRAM headroom. Once you know the plain path works, then add the faster path back in piece by piece.