+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: vLLM / SGLang (https://www.datasculpting.ai/inference/forumdisplay.php?fid=36)
+--- Thread: prefill/decode separation in practice on B200 (/showthread.php?tid=1198)
prefill/decode separation in practice on B200 - cacheline - 10-15-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
This isn't revolutionary, but the combination of PyTorch and M3 Max felt surprisingly usable once I tuned around KV cache pressure.
If someone wants the exact settings I can clean them up and post them.
Re: prefill/decode separation in practice on B200 - thetashed24 - 10-15-2026
Quote:Thought I'd post a small recipe because a lot of threads ask similar things.
Interesting, I hadn't tried that. I ended up in about the same place using LM Studio. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: prefill/decode separation in practice on B200 - signalcraft14 - 10-18-2026
Interesting, I hadn't tried that. I ended up in about the same place using vLLM. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: prefill/decode separation in practice on B200 - vrambandit - 10-18-2026
I tested this tonight. On 5090 with llama.cpp, I saw basically the same behavior. The main thing that moved was latency, not quality.
Re: prefill/decode separation in practice on B200 - amberrunner - 10-18-2026
Interesting, I hadn't tried that. That matches what I saw on Threadripper box. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: prefill/decode separation in practice on B200 - batchsize - 10-18-2026
My first guess would be this: If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: prefill/decode separation in practice on B200 - batchember - 10-18-2026
Maybe, but I had the opposite result. I get why that seems right, but on A6000 the bottleneck felt elsewhere.