when TensorRT-LLM clearly wins on 5090

[kernelmike]

[quote]I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Mixtral and compare from there. A lot of these

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Gemma and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.

[batchcraft]

What exact setup are you running? Do you see the same thing with a smaller batch or shorter prompt?

[amberpatch25]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Mixtral and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[routecrate]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Mixtral and compare from there. A lot of these threads

Interesting, I hadn't tried that. The pattern around KV cache pressure has been pretty similar for me.