+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: vLLM / SGLang (https://www.datasculpting.ai/inference/forumdisplay.php?fid=36)
+--- Thread: router in front of vLLM: anyone testing this? (/showthread.php?tid=65)
Pages:
1
2
router in front of vLLM: anyone testing this? - batchsize - 08-29-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
Just sharing a setup that ended up working reasonably well: Qwen, llama.cpp, Threadripper box, with a lot of attention on router policy.
Maybe useful as a starting point for newer folks.
This is mostly for local work rather than public serving.
The part that surprised me was how much the boring details mattered more than the model choice.
Re: router in front of vLLM: anyone testing this? - benchplease - 08-29-2026
Maybe, but I had the opposite result. I've seen people blame batch size when the real issue was much more mundane.
Re: router in front of vLLM: anyone testing this? - southpatch - 08-30-2026
Quote:Maybe, but I had the opposite result. I've seen people blame batch size when the real issue was much more mundane.
I think the bottleneck is somewhere else. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Could be a workload difference, but I wouldn't generalize too fast from one setup.
Re: router in front of vLLM: anyone testing this? - batchsize - 08-30-2026
worth checking the driver first.
If you post your exact settings, people here can probably narrow it down pretty fast.
Re: router in front of vLLM: anyone testing this? - northhall88 - 08-30-2026
I'd approach it in a boring order. If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious.
Re: router in front of vLLM: anyone testing this? - mangofield33 - 09-02-2026
What worked for me was pretty simple. If you're on Ollama, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.
Re: router in front of vLLM: anyone testing this? - phasestream53 - 09-03-2026
Maybe, but I had the opposite result. I've seen people blame INT4 when the real issue was much more mundane. Could be a workload difference, but I wouldn't generalize too fast from one setup.
Re: router in front of vLLM: anyone testing this? - batchcraft - 09-04-2026
Quote:Maybe, but I had the opposite result. I've seen people blame INT4 when the real issue was much more mundane. Could be a workload difference, but I wou
I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the simpler explanation before adopting the more interesting one.
Re: router in front of vLLM: anyone testing this? - cozysmith - 09-04-2026
My first guess would be this: Before rebuilding anything, I'd test one clean baseline with FLUX and compare from there.
Re: router in front of vLLM: anyone testing this? - batchsize - 09-05-2026
Quote:My first guess would be this: Before rebuilding anything, I'd test one clean baseline with FLUX and compare from there.
Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.