prefill bottlenecks on longer prompts with vLLM

[cudaline]

This keeps coming up in conversation so I figured I'd start a thread.

One thing I keep noticing is that people underestimate continuous batching until they actually try to run this stuff for more than one afternoon.

Curious whether that lines up with what other people are seeing.

[cobaltlab]

This keeps coming up in conversation so I figured I'd start a thread.

What exact setup are you running? Do you see the same thing with a smaller batch or shorter prompt?

[latentscope35]

that fixed it for me too.

[tinycontext]

following this because I'm in the same boat.

[cudaworks]

Ran a quick check on my box. On dual 3090 with PyTorch, I saw basically the same behavior. The main thing that moved was latency, not quality.

[northquill]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.

[vectorfield]

What exact setup are you running? What model, what runtime, and roughly what context size?

[phasescope]

Thanks, that helps. That matches what I saw on Threadripper box.

[thetashed24]

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.

[circuitwire58]

I'd start with the simpler stack.