when TensorRT-LLM clearly wins with MLX

[rasterbench]

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Kimi, TensorRT-LLM, dual 3090, with a lot of attention on reranking.

If someone wants the exact settings I can clean them up and post them.

[pixelcrate82]

That matches what I was seeing. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[northcraft90]

A couple things I'd check first. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[macrotinker]

A couple things I'd check first. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefill/decode split. Once you know the plain path works, then add the faster path back in piece by piece.

I'm happy to test one more variant if you want a second data point.

[kernelmike]

Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. I expected the model choice to matter more than it did.

[syshouse]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Llama and compare from there.

[kernelmike]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Llama and compare from there.

Good point. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.