CUDA update broke my throughput for Kimi

[kernelmike]

Spent some time on this today and wanted to compare notes.

Two paths both seem viable, but they break down differently once quant drift becomes important.

Trying to avoid rebuilding the whole thing twice.

I care more about stability than squeezing the last possible tokens/sec out of it.

[kernelmike]

Spent some time on this today and wanted to compare notes.

Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did. Turns out 'just one more test' was a lie.

[nanopatch]

I tested this tonight. On 7900 XTX with Ollama, I saw basically the same behavior.

[edgecoast]

I tested this tonight. On Threadripper box with TensorRT-LLM, I saw basically the same behavior. I expected the model choice to matter more than it did.

[amberpatch25]

That matches what I was seeing. The pattern around structured output has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[dockember]

worth checking the driver first.

[phaseport36]

Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?

[kernelmike]

Can you fill in a bit more context? Are you testing this on Threadripper box or something else?

[phaseport36]

My first guess would be this: If you're on MLX, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.

[orbitbench6]

I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. I expected the model choice to matter more than it did. Turns out 'just one more test' was a lie.