+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: vLLM / SGLang (https://www.datasculpting.ai/inference/forumdisplay.php?fid=36)
+--- Thread: prefill/decode separation in practice for Kimi (/showthread.php?tid=1041)
Pages:
1
2
prefill/decode separation in practice for Kimi - batchsize - 10-11-2026
Wanted to compare notes with people doing similar things.
I'm mostly working with Phi in llama.cpp on B200. The part I'm trying to understand is tool calling.
If you've tested something similar, what did you keep and what did you stop bothering with?
If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.
Re: prefill/decode separation in practice for Kimi - rocmrunner40 - 10-11-2026
Quote:Wanted to compare notes with people doing similar things.
Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?
Re: prefill/decode separation in practice for Kimi - promptpoint - 10-12-2026
Not sure I agree there. I get why that seems right, but on B200 the bottleneck felt elsewhere. Could be a workload difference, but I wouldn't generalize too fast from one setup.
Re: prefill/decode separation in practice for Kimi - signalcraft14 - 10-12-2026
I hit almost the same issue last night.
Re: prefill/decode separation in practice for Kimi - batchsize - 10-13-2026
I tested this tonight. On 5090 with PyTorch, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.
Re: prefill/decode separation in practice for Kimi - routearch - 10-13-2026
That matches what I was seeing. That matches what I saw on 5090. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: prefill/decode separation in practice for Kimi - turboarch95 - 10-13-2026
Appreciate the detail. That matches what I saw on 5090. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: prefill/decode separation in practice for Kimi - mangofield33 - 10-13-2026
Quote:Appreciate the detail. That matches what I saw on 5090. Worth rechecking your defaults though, because one bad assumption can make the rest of the thr
I'd start with the simpler stack.
If you post your exact settings, people here can probably narrow it down pretty fast.
Re: prefill/decode separation in practice for Kimi - rocmmixer - 10-14-2026
subscribed, curious where this lands.
Re: prefill/decode separation in practice for Kimi - mangofield33 - 10-17-2026
I hit almost the same issue last night.