+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Drivers / Frameworks / Toolkits (https://www.datasculpting.ai/inference/forumdisplay.php?fid=8)
+--- Forum: Apple / MLX (https://www.datasculpting.ai/inference/forumdisplay.php?fid=33)
+--- Thread: MLX model loading quirks (/showthread.php?tid=1353)
Pages:
1
2
MLX model loading quirks - rocmrunner40 - 10-17-2026
Has anyone else run into this?
I'm mostly working with DeepSeek in llama.cpp on Threadripper box. The part I'm trying to understand is batch size.
Not looking for the perfect answer, mostly what worked in real use.
Re: MLX model loading quirks - pixelcrate82 - 10-17-2026
What worked for me was pretty simple. On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.
Re: MLX model loading quirks - thetaarch43 - 10-18-2026
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at batch size.
Re: MLX model loading quirks - quietarch - 10-19-2026
Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly.
Re: MLX model loading quirks - labnest - 10-19-2026
My first guess would be this: If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: MLX model loading quirks - plainember52 - 10-19-2026
Appreciate the detail. I ended up in about the same place using LM Studio. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: MLX model loading quirks - torchshed - 10-19-2026
Quote:My first guess would be this: If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot
Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?
Re: MLX model loading quirks - dockridge - 10-19-2026
Interesting, I hadn't tried that. The pattern around tokens/sec has been pretty similar for me.
Re: MLX model loading quirks - mlxgarden - 10-19-2026
Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal.
Re: MLX model loading quirks - sanescope - 10-19-2026
Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.