+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Drivers / Frameworks / Toolkits (https://www.datasculpting.ai/inference/forumdisplay.php?fid=8)
+--- Forum: NVIDIA / CUDA (https://www.datasculpting.ai/inference/forumdisplay.php?fid=31)
+--- Thread: CUDA update broke my throughput (/showthread.php?tid=184)
Pages:
1
2
CUDA update broke my throughput - amberpatch25 - 09-08-2026
Did a bit of testing and thought I'd post the result here.
Short version: TensorRT-LLM behaved well for me, but only after I stopped ignoring prefill/decode split.
Curious if anyone else landed in the same place.
For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.
Re: CUDA update broke my throughput - dockember - 09-08-2026
Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal.
Re: CUDA update broke my throughput - cobaltspark - 09-08-2026
My first guess would be this: If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.
If you post your exact settings, people here can probably narrow it down pretty fast.
Re: CUDA update broke my throughput - phaseport36 - 09-08-2026
That matches what I was seeing. That matches what I saw on dual 3090. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: CUDA update broke my throughput - cobaltspark - 09-09-2026
Quote:That matches what I was seeing. That matches what I saw on dual 3090. Worth rechecking your defaults though, because one bad assumption can make the r
I tested this tonight. On 5090 with vLLM, I saw basically the same behavior. I expected the model choice to matter more than it did.
Re: CUDA update broke my throughput - phaseport36 - 09-09-2026
I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at router policy. Not glamorous advice, but repeatable baselines save a lot of time.
Re: CUDA update broke my throughput - audiopath66 - 09-11-2026
Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. I expected the model choice to matter more than it did.
Re: CUDA update broke my throughput - amberpatch25 - 09-11-2026
Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.
Re: CUDA update broke my throughput - phaseport36 - 09-13-2026
that fixed it for me too.
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.
Re: CUDA update broke my throughput - amberpatch25 - 09-13-2026
that fixed it for me too.