TensorRT-LLM worth the setup time

[kernelmike]

I figured I'd ask here before changing everything.

I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling reranking.

Would appreciate practical advice rather than theory.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

[routecrate]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tokens/sec.

[kernelmike]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tokens/sec.

that fixed it for me too.

[routecrate]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at reranking.

[kernelmike]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefill/decode split. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[audiodesk]

I'd approach it in a boring order. On M3 Max, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[thetapath]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at embeddings.

[vectorport]

I hit almost the same issue last night.

[torchsmith25]

Interesting, I hadn't tried that. I don't think your conclusion is far off.

[tensorfield88]

That matches what I was seeing. The pattern around structured output has been pretty similar for me.