why did throughput drop after update for Kimi

[labpath]

I'm trying to sanity-check my setup.

I'm mostly working with Qwen in TensorRT-LLM on A6000. The part I'm trying to understand is speculative decoding.

Would appreciate practical advice rather than theory.

[audioforge]

Good point. That matches what I saw on GB200. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[visionforge]

Good point. That matches what I saw on GB200. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look c

I'd approach it in a boring order. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.

[modrunner27]

What exact setup are you running? Are you testing this on dual 3090 or something else?

I'm happy to test one more variant if you want a second data point.

[turboarch95]

That matches what I was seeing. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[nanogarden]

Can you fill in a bit more context? What model, what runtime, and roughly what context size?

[oxideworks59]

Good point. That matches what I saw on Threadripper box. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[thetashed24]

I hit almost the same issue last night.

[thetaplane]

same here on my side.

[signalcraft14]

I think the bottleneck is somewhere else. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Worth testing the simpler explanation before adopting the more interesting one.