+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: vLLM / SGLang (https://www.datasculpting.ai/inference/forumdisplay.php?fid=36)
+--- Thread: continuous batching wins and losses with Ollama (/showthread.php?tid=652)
Pages:
1
2
continuous batching wins and losses with Ollama - oxidemixer - 10-01-2026
Spent some time on this today and wanted to compare notes.
I've gone back and forth between Ollama and LM Studio for Qwen. The interesting difference for me has been speculative decoding.
Trying to avoid rebuilding the whole thing twice.
Re: continuous batching wins and losses with Ollama - routearch - 10-03-2026
I don't think that's the whole story. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Could be a workload difference, but I wouldn't generalize too fast from one setup.
Re: continuous batching wins and losses with Ollama - batchsize - 10-06-2026
Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.
Re: continuous batching wins and losses with Ollama - audioforge - 10-06-2026
same here on my side.
Re: continuous batching wins and losses with Ollama - phasestream53 - 10-06-2026
subscribed, curious where this lands.
Re: continuous batching wins and losses with Ollama - batchsize - 10-06-2026
Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.
Re: continuous batching wins and losses with Ollama - bytegarden73 - 10-07-2026
I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.
Re: continuous batching wins and losses with Ollama - routecrate - 10-07-2026
Good point. I ended up in about the same place using SGLang.
Re: continuous batching wins and losses with Ollama - pixelcrate82 - 10-07-2026
Quote:Good point. I ended up in about the same place using SGLang.
Maybe, but I had the opposite result. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the simpler explanation before adopting the more interesting one.
Re: continuous batching wins and losses with Ollama - amberrunner - 10-08-2026
I don't think that's the whole story. I've seen people blame tail latency when the real issue was much more mundane. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.
If you post your exact settings, people here can probably narrow it down pretty fast.