+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: vLLM / SGLang (https://www.datasculpting.ai/inference/forumdisplay.php?fid=36)
+--- Thread: continuous batching wins and losses (/showthread.php?tid=1111)
Pages:
1
2
Re: continuous batching wins and losses - promptpoint - 10-16-2026
Appreciate the detail. The pattern around tail latency has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: continuous batching wins and losses - batchsize - 10-16-2026
Thanks, that helps. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: continuous batching wins and losses - ggufvault - 10-17-2026
Can you fill in a bit more context? Is this local-only or are you routing anything externally?
Re: continuous batching wins and losses - nanogarden - 10-17-2026
Need a little more detail here. Are you testing this on 7900 XTX or something else?
Re: continuous batching wins and losses - mangofield33 - 10-19-2026
Quote:Need a little more detail here. Are you testing this on 7900 XTX or something else?
Maybe, but I had the opposite result. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Could be a workload difference, but I wouldn't generalize too fast from one setup.