+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: vLLM / SGLang (https://www.datasculpting.ai/inference/forumdisplay.php?fid=36)
+--- Thread: router in front of vLLM for Gemma (/showthread.php?tid=376)
router in front of vLLM for Gemma - batchsize - 09-20-2026
Maybe I'm missing something obvious.
My current setup is llama.cpp with GB200, and I keep running into prefill/decode split.
Not looking for the perfect answer, mostly what worked in real use.
For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.
If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.
Re: router in front of vLLM for Gemma - nanogarden - 09-20-2026
subscribed, curious where this lands.
Re: router in front of vLLM for Gemma - northgarden37 - 09-23-2026
Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the simpler explanation before adopting the more interesting one.
Re: router in front of vLLM for Gemma - batchcraft - 09-25-2026
My first guess would be this: If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.
Re: router in front of vLLM for Gemma - promptpoint - 09-26-2026
What worked for me was pretty simple. If you're on MLX, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.
Re: router in front of vLLM for Gemma - cacheline - 09-26-2026
I don't think that's the whole story. I get why that seems right, but on 5090 the bottleneck felt elsewhere. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.
Re: router in front of vLLM for Gemma - signalpeak - 09-26-2026
Quote:I don't think that's the whole story. I get why that seems right, but on 5090 the bottleneck felt elsewhere. Not saying you're wrong, just that I woul
A couple things I'd check first. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.