model merge got better or just different

[thetatinker77]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Mixtral and compare from there.

[bytestack]

That matches what I was seeing. I ended up in about the same place using MLX.

[mangohouse]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at MoE serving. Once you know the plain path works, then add the faster path back in piece by piece.

[rocmstack25]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at MoE serving. Once you kn

Ran a quick check on my box. The issue went away once I changed one setting and restarted the stack cleanly.

[cacheline]

What worked for me was pretty simple. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.