+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Prompting & Context Engineering (https://www.datasculpting.ai/inference/forumdisplay.php?fid=15)
+--- Forum: Prompts / System Prompts (https://www.datasculpting.ai/inference/forumdisplay.php?fid=62)
+--- Thread: prompt length getting silly (/showthread.php?tid=1118)
Pages:
1
2
prompt length getting silly - fusedhall56 - 10-12-2026
Spent some time on this today and wanted to compare notes.
I've gone back and forth between llama.cpp and vLLM for Gemma. The interesting difference for me has been speculative decoding.
Trying to avoid rebuilding the whole thing twice.
I care more about stability than squeezing the last possible tokens/sec out of it.
Re: prompt length getting silly - mintforge - 10-12-2026
Need a little more detail here. Is this local-only or are you routing anything externally?
Re: prompt length getting silly - bytelab - 10-13-2026
subscribed, curious where this lands.
Re: prompt length getting silly - thetaarch43 - 10-13-2026
A couple things I'd check first. On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Re: prompt length getting silly - promptspark58 - 10-13-2026
Not sure I agree there. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.
Re: prompt length getting silly - batchember - 10-13-2026
Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher.
Re: prompt length getting silly - mangoquill - 10-16-2026
That matches what I was seeing. The pattern around tokens/sec has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: prompt length getting silly - rasterhall - 10-17-2026
Quote:That matches what I was seeing. The pattern around tokens/sec has been pretty similar for me. I got better results once I stopped chasing one metric a
I'd approach it in a boring order. On B200, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: prompt length getting silly - thetaember38 - 10-17-2026
I think the bottleneck is somewhere else. I've seen people blame long context when the real issue was much more mundane.
Re: prompt length getting silly - thetaarch43 - 10-18-2026
A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?