MLX model loading quirks

[pixelcrate82]

Maybe I'm missing something obvious.

My current setup is TensorRT-LLM with B200, and I keep running into speculative decoding.

If you've tested something similar, what did you keep and what did you stop bothering with?

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

[mellowstream]

A couple questions before I guess too hard. Is this local-only or are you routing anything externally?

If you post your exact settings, people here can probably narrow it down pretty fast.

[sanescope]

subscribed, curious where this lands.

[mlxgarden]

A couple things I'd check first. If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Turns out 'just one more test' was a lie.

[circuitwire58]

What worked for me was pretty simple. If you're on MLX, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[thetaarch43]

Can you fill in a bit more context? Are you testing this on B200 or something else?

[dockridge]

That matches what I was seeing. I ended up in about the same place using vLLM.