best small quant that still feels smart for Kimi

[cachebench]

Spent some time on this today and wanted to compare notes.

I've gone back and forth between TensorRT-LLM and vLLM for Kimi. The interesting difference for me has been KV cache pressure.

Interested in tradeoffs more than fan loyalty.

[benchplease]

Spent some time on this today and wanted to compare notes.

Need a little more detail here. Is this local-only or are you routing anything externally?

[mangofield33]

Interesting, I hadn't tried that. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[vrambandit]

What worked for me was pretty simple. On A6000, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.

[benchplease]

that fixed it for me too.

[relaynest82]

Tried to reproduce it just now. On M3 Max with SGLang, I saw basically the same behavior. The main thing that moved was latency, not quality.

[nanobrook]

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Llama and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.

[cudaline]

I think the bottleneck is somewhere else. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.