MLX model loading quirks

[rocmrunner40]

Has anyone else run into this?

I'm mostly working with DeepSeek in llama.cpp on Threadripper box. The part I'm trying to understand is batch size.

Not looking for the perfect answer, mostly what worked in real use.

[pixelcrate82]

What worked for me was pretty simple. On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.

[thetaarch43]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at batch size.

[quietarch]

Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly.

[labnest]

My first guess would be this: If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[plainember52]

Appreciate the detail. I ended up in about the same place using LM Studio. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[torchshed]

My first guess would be this: If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot

Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?

[dockridge]

Interesting, I hadn't tried that. The pattern around tokens/sec has been pretty similar for me.

[mlxgarden]

Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal.

[sanescope]

Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.