+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Optimization & Performance (https://www.datasculpting.ai/inference/forumdisplay.php?fid=10)
+--- Forum: Performance Tuning (https://www.datasculpting.ai/inference/forumdisplay.php?fid=42)
+--- Thread: what changed after the inference shift for Kimi (/showthread.php?tid=882)
Pages:
1
2
what changed after the inference shift for Kimi - edgeriver - 10-07-2026
Posting this in case it saves someone else a little time.
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about small local stacks.
I'm especially interested in mistakes worth avoiding.
Re: what changed after the inference shift for Kimi - bytestack - 10-09-2026
Not sure I agree there. I get why that seems right, but on A6000 the bottleneck felt elsewhere.
Re: what changed after the inference shift for Kimi - tensorfield88 - 10-09-2026
Interesting, I hadn't tried that. The pattern around token cost has been pretty similar for me.
Re: what changed after the inference shift for Kimi - localpatch - 10-09-2026
I'd approach it in a boring order. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.
Re: what changed after the inference shift for Kimi - stackarch - 10-10-2026
Not sure I agree there. I get why that seems right, but on A6000 the bottleneck felt elsewhere. Could be a workload difference, but I wouldn't generalize too fast from one setup.
Re: what changed after the inference shift for Kimi - vramport - 10-10-2026
I don't think that's the whole story. I've seen people blame tool calling when the real issue was much more mundane. Worth testing the simpler explanation before adopting the more interesting one.
Re: what changed after the inference shift for Kimi - prompttinker44 - 10-11-2026
Maybe, but I had the opposite result. I get why that seems right, but on GB200 the bottleneck felt elsewhere. Could be a workload difference, but I wouldn't generalize too fast from one setup.
Re: what changed after the inference shift for Kimi - macropilot - 10-12-2026
I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Phi and compare from there.
Re: what changed after the inference shift for Kimi - papermixer4 - 10-12-2026
A couple things I'd check first. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious.
Re: what changed after the inference shift for Kimi - cudapeak - 10-13-2026
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at KV cache pressure.