why did throughput drop after update

[batchsize]

Posting this in case it saves someone else a little time.

I'm not trying to build a perfect architecture diagram. I just want a sane starting point around vLLM and 4090.

Would rather learn from your scars than collect my own all at once.

I care more about stability than squeezing the last possible tokens/sec out of it.

The part that surprised me was how much the boring details mattered more than the model choice.

[signalcraft14]

What exact setup are you running? Do you see the same thing with a smaller batch or shorter prompt?

[batchsize]

My first guess would be this: If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.

[cacheline]

My first guess would be this: If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorou

subscribed, curious where this lands.

[mangofield33]

I don't think that's the whole story. I get why that seems right, but on B200 the bottleneck felt elsewhere. Worth testing the simpler explanation before adopting the more interesting one.

[nanogarden]

Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal.

[audioforge]

I'd approach it in a boring order. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[batchsize]

that fixed it for me too.

If you post your exact settings, people here can probably narrow it down pretty fast.

[ggufvault]

I tested this tonight. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.

[batchsize]

Need a little more detail here. What model, what runtime, and roughly what context size?