+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Drivers / Frameworks / Toolkits (https://www.datasculpting.ai/inference/forumdisplay.php?fid=8)
+--- Forum: NVIDIA / CUDA (https://www.datasculpting.ai/inference/forumdisplay.php?fid=31)
+--- Thread: CUDA update broke my throughput (/showthread.php?tid=939)
Pages:
1
2
CUDA update broke my throughput - mintforge - 10-10-2026
Posting this in case it saves someone else a little time.
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about INT4.
Would rather learn from your scars than collect my own all at once.
Re: CUDA update broke my throughput - kernelmike - 10-11-2026
Not sure I agree there. I get why that seems right, but on A6000 the bottleneck felt elsewhere. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.
Re: CUDA update broke my throughput - thetapath - 10-11-2026
Appreciate the detail. The pattern around prefix caching has been pretty similar for me.
Re: CUDA update broke my throughput - kernelmike - 10-11-2026
Quote:Appreciate the detail. The pattern around prefix caching has been pretty similar for me.
same here on my side.
Re: CUDA update broke my throughput - dockember - 10-12-2026
Interesting, I hadn't tried that. The pattern around router policy has been pretty similar for me.
Re: CUDA update broke my throughput - audiodesk - 10-14-2026
What worked for me was pretty simple. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.
If you post your exact settings, people here can probably narrow it down pretty fast.
Re: CUDA update broke my throughput - batchsize - 10-14-2026
That matches what I was seeing. The pattern around router policy has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
I'm happy to test one more variant if you want a second data point.
Re: CUDA update broke my throughput - cozysmith - 10-14-2026
Not sure I agree there. I get why that seems right, but on mini PC the bottleneck felt elsewhere.
Re: CUDA update broke my throughput - macrostack69 - 10-16-2026
Not sure I agree there. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Could be a workload difference, but I wouldn't generalize too fast from one setup.
Re: CUDA update broke my throughput - amberpatch25 - 10-16-2026
Good point. The pattern around batch size has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.