+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Drivers / Frameworks / Toolkits (https://www.datasculpting.ai/inference/forumdisplay.php?fid=8)
+--- Forum: NVIDIA / CUDA (https://www.datasculpting.ai/inference/forumdisplay.php?fid=31)
+--- Thread: CUDA update broke my throughput for Kimi (/showthread.php?tid=704)
Pages:
1
2
CUDA update broke my throughput for Kimi - kernelmike - 10-03-2026
Spent some time on this today and wanted to compare notes.
Two paths both seem viable, but they break down differently once quant drift becomes important.
Trying to avoid rebuilding the whole thing twice.
I care more about stability than squeezing the last possible tokens/sec out of it.
Re: CUDA update broke my throughput for Kimi - kernelmike - 10-05-2026
Quote:Spent some time on this today and wanted to compare notes.
Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did. Turns out 'just one more test' was a lie.
Re: CUDA update broke my throughput for Kimi - nanopatch - 10-05-2026
I tested this tonight. On 7900 XTX with Ollama, I saw basically the same behavior.
Re: CUDA update broke my throughput for Kimi - edgecoast - 10-06-2026
I tested this tonight. On Threadripper box with TensorRT-LLM, I saw basically the same behavior. I expected the model choice to matter more than it did.
Re: CUDA update broke my throughput for Kimi - amberpatch25 - 10-07-2026
That matches what I was seeing. The pattern around structured output has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: CUDA update broke my throughput for Kimi - dockember - 10-07-2026
worth checking the driver first.
Re: CUDA update broke my throughput for Kimi - phaseport36 - 10-10-2026
Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?
Re: CUDA update broke my throughput for Kimi - kernelmike - 10-10-2026
Can you fill in a bit more context? Are you testing this on Threadripper box or something else?
Re: CUDA update broke my throughput for Kimi - phaseport36 - 10-11-2026
My first guess would be this: If you're on MLX, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.
Re: CUDA update broke my throughput for Kimi - orbitbench6 - 10-11-2026
I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. I expected the model choice to matter more than it did. Turns out 'just one more test' was a lie.