best local open-weight model this week for Kimi

[amberhall]

This keeps coming up in conversation so I figured I'd start a thread.

One thing I keep noticing is that people underestimate KV cache pressure until they actually try to run this stuff for more than one afternoon.

Curious whether that lines up with what other people are seeing.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

[northquill]

Appreciate the detail. That matches what I saw on A6000. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[kernelmike]

Need a little more detail here. Do you see the same thing with a smaller batch or shorter prompt?

[latentpeak22]

worth checking the driver first.

[southpatch]

worth checking the driver first.

[torchtrack24]

worth checking the driver first.

Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.

[amberpatch25]

Appreciate the detail. I ended up in about the same place using LM Studio. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[cobaltspark]

following this because I'm in the same boat.

[tinycontext]

same here on my side.

[fastfield4]

What worked for me was pretty simple. On A6000, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.