Inference
odd little inference appliances with MLX - Printable Version

+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Edge AI & Gadgets (https://www.datasculpting.ai/inference/forumdisplay.php?fid=21)
+--- Forum: AI Devices (https://www.datasculpting.ai/inference/forumdisplay.php?fid=75)
+--- Thread: odd little inference appliances with MLX (/showthread.php?tid=781)

odd little inference appliances with MLX - mintquill - 10-05-2026

Did a bit of testing and thought I'd post the result here.

After a few runs on 7900 XTX, I ended up with decent performance, but the setup only really made sense once I adjusted around small local stacks.

Happy to post more details if anyone wants them.

This is mostly for local work rather than public serving.

Re: odd little inference appliances with MLX - flintriver50 - 10-06-2026

same here on my side.

Re: odd little inference appliances with MLX - latentscope35 - 10-09-2026

Appreciate the detail. I ended up in about the same place using PyTorch. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

Re: odd little inference appliances with MLX - mangopath4 - 10-11-2026

Quote:Appreciate the detail. I ended up in about the same place using PyTorch. The biggest change for me was simplifying the stack instead of adding one mor

A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?

Re: odd little inference appliances with MLX - routecrate - 10-11-2026

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at VRAM headroom.

Re: odd little inference appliances with MLX - bytepilot - 10-12-2026

Quote:I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at VRAM headroom.

I don't think that's the whole story. I've seen people blame structured output when the real issue was much more mundane. Worth testing the simpler explanation before adopting the more interesting one.

Re: odd little inference appliances with MLX - flintriver50 - 10-15-2026

Thanks, that helps. That matches what I saw on GB200.

Re: odd little inference appliances with MLX - audiorunner74 - 10-16-2026

Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.