where open models feel surprisingly good

[batchcraft]

What worked for me was pretty simple. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[cacheplane98]

Good point. That matches what I saw on 7900 XTX.

[mlxgarden]

Interesting, I hadn't tried that. That matches what I saw on A6000.

[deltagrid]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at long context. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.