+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: llama.cpp / GGUF (https://www.datasculpting.ai/inference/forumdisplay.php?fid=35)
+--- Thread: context size vs speed in GGUF with vLLM (/showthread.php?tid=1045)
Pages:
1
2
context size vs speed in GGUF with vLLM - embershed99 - 10-12-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
This isn't revolutionary, but the combination of PyTorch and B200 felt surprisingly usable once I tuned around consumer GPUs.
Feel free to tear it apart if you see something dumb.
Re: context size vs speed in GGUF with vLLM - vrambandit - 10-13-2026
Good point. That matches what I saw on 4090. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: context size vs speed in GGUF with vLLM - vectorport - 10-16-2026
I hit almost the same issue last night.
Re: context size vs speed in GGUF with vLLM - audioriver - 10-16-2026
Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.
Re: context size vs speed in GGUF with vLLM - mangotinker73 - 10-18-2026
I'd start with the simpler stack.
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.
Re: context size vs speed in GGUF with vLLM - mellowwire98 - 10-18-2026
Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal.
Re: context size vs speed in GGUF with vLLM - pixelcrate82 - 10-18-2026
I tested this tonight. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.
Re: context size vs speed in GGUF with vLLM - cudacrate - 10-18-2026
Quote:I'd start with the simpler stack.
Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher. I expected the model choice to matter more than it did.
Re: context size vs speed in GGUF with vLLM - mangotinker73 - 10-18-2026
Interesting, I hadn't tried that. I ended up in about the same place using PyTorch. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
I'm happy to test one more variant if you want a second data point.
Re: context size vs speed in GGUF with vLLM - latentcraft - 10-18-2026
A couple things I'd check first. On 4090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.
I'm happy to test one more variant if you want a second data point.