prompt length getting silly

[fusedhall56]

Spent some time on this today and wanted to compare notes.

I've gone back and forth between llama.cpp and vLLM for Gemma. The interesting difference for me has been speculative decoding.

Trying to avoid rebuilding the whole thing twice.

I care more about stability than squeezing the last possible tokens/sec out of it.

[mintforge]

Need a little more detail here. Is this local-only or are you routing anything externally?

[bytelab]

subscribed, curious where this lands.

[thetaarch43]

A couple things I'd check first. On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[promptspark58]

Not sure I agree there. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[batchember]

Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher.

[mangoquill]

That matches what I was seeing. The pattern around tokens/sec has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[rasterhall]

That matches what I was seeing. The pattern around tokens/sec has been pretty similar for me. I got better results once I stopped chasing one metric a

I'd approach it in a boring order. On B200, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[thetaember38]

I think the bottleneck is somewhere else. I've seen people blame long context when the real issue was much more mundane.

[thetaarch43]

A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?