what changed after the inference shift for Kimi

[edgeriver]

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about small local stacks.

I'm especially interested in mistakes worth avoiding.

[bytestack]

Not sure I agree there. I get why that seems right, but on A6000 the bottleneck felt elsewhere.

[tensorfield88]

Interesting, I hadn't tried that. The pattern around token cost has been pretty similar for me.

[localpatch]

I'd approach it in a boring order. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.

[stackarch]

Not sure I agree there. I get why that seems right, but on A6000 the bottleneck felt elsewhere. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[vramport]

I don't think that's the whole story. I've seen people blame tool calling when the real issue was much more mundane. Worth testing the simpler explanation before adopting the more interesting one.

[prompttinker44]

Maybe, but I had the opposite result. I get why that seems right, but on GB200 the bottleneck felt elsewhere. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[macropilot]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Phi and compare from there.

[papermixer4]

A couple things I'd check first. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious.

[cudapeak]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at KV cache pressure.