MLX model loading quirks

[thetaarch43]

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Gemma, vLLM, mini PC, with a lot of attention on consumer GPUs.

Feel free to tear it apart if you see something dumb.

[plainember52]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at structured output. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.