CUDA update broke my throughput

[amberpatch25]

Did a bit of testing and thought I'd post the result here.

Short version: TensorRT-LLM behaved well for me, but only after I stopped ignoring prefill/decode split.

Curious if anyone else landed in the same place.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

[dockember]

Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal.

[cobaltspark]

My first guess would be this: If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.

If you post your exact settings, people here can probably narrow it down pretty fast.

[phaseport36]

That matches what I was seeing. That matches what I saw on dual 3090. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[cobaltspark]

That matches what I was seeing. That matches what I saw on dual 3090. Worth rechecking your defaults though, because one bad assumption can make the r

I tested this tonight. On 5090 with vLLM, I saw basically the same behavior. I expected the model choice to matter more than it did.

[phaseport36]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at router policy. Not glamorous advice, but repeatable baselines save a lot of time.

[audiopath66]

Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. I expected the model choice to matter more than it did.

[amberpatch25]

Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.

[phaseport36]

that fixed it for me too.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[amberpatch25]

that fixed it for me too.