+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: Other Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=39)
+--- Thread: prefill bottlenecks on longer prompts with vLLM (/showthread.php?tid=9)
Pages:
1
2
prefill bottlenecks on longer prompts with vLLM - cudaline - 08-23-2026
This keeps coming up in conversation so I figured I'd start a thread.
One thing I keep noticing is that people underestimate continuous batching until they actually try to run this stuff for more than one afternoon.
Curious whether that lines up with what other people are seeing.
Re: prefill bottlenecks on longer prompts with vLLM - cobaltlab - 08-23-2026
Quote:This keeps coming up in conversation so I figured I'd start a thread.
What exact setup are you running? Do you see the same thing with a smaller batch or shorter prompt?
Re: prefill bottlenecks on longer prompts with vLLM - latentscope35 - 08-23-2026
that fixed it for me too.
Re: prefill bottlenecks on longer prompts with vLLM - tinycontext - 08-24-2026
following this because I'm in the same boat.
Re: prefill bottlenecks on longer prompts with vLLM - cudaworks - 08-24-2026
Ran a quick check on my box. On dual 3090 with PyTorch, I saw basically the same behavior. The main thing that moved was latency, not quality.
Re: prefill bottlenecks on longer prompts with vLLM - northquill - 08-24-2026
My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.
Re: prefill bottlenecks on longer prompts with vLLM - vectorfield - 08-25-2026
What exact setup are you running? What model, what runtime, and roughly what context size?
Re: prefill bottlenecks on longer prompts with vLLM - phasescope - 08-26-2026
Thanks, that helps. That matches what I saw on Threadripper box.
Re: prefill bottlenecks on longer prompts with vLLM - thetashed24 - 08-26-2026
What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.
Re: prefill bottlenecks on longer prompts with vLLM - circuitwire58 - 08-26-2026
I'd start with the simpler stack.