odd little inference appliances with MLX

[mintquill]

Did a bit of testing and thought I'd post the result here.

After a few runs on 7900 XTX, I ended up with decent performance, but the setup only really made sense once I adjusted around small local stacks.

Happy to post more details if anyone wants them.

This is mostly for local work rather than public serving.

[flintriver50]

same here on my side.

[latentscope35]

Appreciate the detail. I ended up in about the same place using PyTorch. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[mangopath4]

Appreciate the detail. I ended up in about the same place using PyTorch. The biggest change for me was simplifying the stack instead of adding one mor

A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?

[routecrate]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at VRAM headroom.

[bytepilot]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at VRAM headroom.

I don't think that's the whole story. I've seen people blame structured output when the real issue was much more mundane. Worth testing the simpler explanation before adopting the more interesting one.

[flintriver50]

Thanks, that helps. That matches what I saw on GB200.

[audiorunner74]

Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.