+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: vLLM / SGLang (https://www.datasculpting.ai/inference/forumdisplay.php?fid=36)
+--- Thread: continuous batching wins and losses with SGLang (/showthread.php?tid=1404)
continuous batching wins and losses with SGLang - cacheline - 10-17-2026
This keeps coming up in conversation so I figured I'd start a thread.
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with INT4.
Curious whether that lines up with what other people are seeing.
Re: continuous batching wins and losses with SGLang - quantyard - 10-18-2026
I hit almost the same issue last night.
Re: continuous batching wins and losses with SGLang - thetaplane - 10-18-2026
That matches what I was seeing. I ended up in about the same place using TensorRT-LLM. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: continuous batching wins and losses with SGLang - audioforge - 10-18-2026
What worked for me was pretty simple. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
I'm happy to test one more variant if you want a second data point.
Re: continuous batching wins and losses with SGLang - rocmrunner40 - 10-19-2026
Quote:I hit almost the same issue last night.
Appreciate the detail. That matches what I saw on B200. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: continuous batching wins and losses with SGLang - batchsize - 10-19-2026
Interesting, I hadn't tried that. The pattern around small local stacks has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: continuous batching wins and losses with SGLang - cozykite - 10-19-2026
My first guess would be this: If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.
If you post your exact settings, people here can probably narrow it down pretty fast.
Re: continuous batching wins and losses with SGLang - phasestream53 - 10-19-2026
A couple things I'd check first. If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: continuous batching wins and losses with SGLang - thetashed24 - 10-19-2026
Quote:A couple things I'd check first. If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of
Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.