+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: vLLM / SGLang (https://www.datasculpting.ai/inference/forumdisplay.php?fid=36)
+--- Thread: continuous batching wins and losses (/showthread.php?tid=1111)
Pages:
1
2
continuous batching wins and losses - routearch - 10-12-2026
Posting this in case it saves someone else a little time.
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about prefix caching.
I'm especially interested in mistakes worth avoiding.
This is mostly for local work rather than public serving.
Re: continuous batching wins and losses - amberplane - 10-14-2026
Quote:Posting this in case it saves someone else a little time.
What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at quant drift.
Re: continuous batching wins and losses - oxidemixer - 10-14-2026
Interesting, I hadn't tried that. The pattern around consumer GPUs has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: continuous batching wins and losses - cacheline - 10-14-2026
Quote:Interesting, I hadn't tried that. The pattern around consumer GPUs has been pretty similar for me. Worth rechecking your defaults though, because one
worth checking the driver first.
Re: continuous batching wins and losses - audioforge - 10-14-2026
Good point. I ended up in about the same place using Ollama. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: continuous batching wins and losses - cacheline - 10-14-2026
I think the bottleneck is somewhere else. I've seen people blame HBM constraints when the real issue was much more mundane. Worth testing the simpler explanation before adopting the more interesting one.
Re: continuous batching wins and losses - oxidemixer - 10-16-2026
Quote:I think the bottleneck is somewhere else. I've seen people blame HBM constraints when the real issue was much more mundane. Worth testing the simpler
A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Kimi and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.
Re: continuous batching wins and losses - deltapoint17 - 10-16-2026
subscribed, curious where this lands.
Re: continuous batching wins and losses - rocmrunner40 - 10-16-2026
subscribed, curious where this lands.
Re: continuous batching wins and losses - signalpeak - 10-16-2026
What exact setup are you running? Do you see the same thing with a smaller batch or shorter prompt?