+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Drivers / Frameworks / Toolkits (https://www.datasculpting.ai/inference/forumdisplay.php?fid=8)
+--- Forum: Apple / MLX (https://www.datasculpting.ai/inference/forumdisplay.php?fid=33)
+--- Thread: MLX model loading quirks (/showthread.php?tid=431)
MLX model loading quirks - pixelcrate82 - 09-22-2026
Maybe I'm missing something obvious.
My current setup is TensorRT-LLM with B200, and I keep running into speculative decoding.
If you've tested something similar, what did you keep and what did you stop bothering with?
My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.
Re: MLX model loading quirks - mellowstream - 09-22-2026
A couple questions before I guess too hard. Is this local-only or are you routing anything externally?
If you post your exact settings, people here can probably narrow it down pretty fast.
Re: MLX model loading quirks - sanescope - 09-25-2026
subscribed, curious where this lands.
Re: MLX model loading quirks - mlxgarden - 09-25-2026
A couple things I'd check first. If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Turns out 'just one more test' was a lie.
Re: MLX model loading quirks - circuitwire58 - 09-26-2026
What worked for me was pretty simple. If you're on MLX, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: MLX model loading quirks - thetaarch43 - 09-27-2026
Can you fill in a bit more context? Are you testing this on B200 or something else?
Re: MLX model loading quirks - dockridge - 09-27-2026
That matches what I was seeing. I ended up in about the same place using vLLM.