+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: llama.cpp / GGUF (https://www.datasculpting.ai/inference/forumdisplay.php?fid=35)
+--- Thread: context size vs speed in GGUF: worth it? (/showthread.php?tid=1487)
Pages:
1
2
context size vs speed in GGUF: worth it? - visionyard23 - 10-18-2026
Spent some time on this today and wanted to compare notes.
I've gone back and forth between Ollama and TensorRT-LLM for Mistral. The interesting difference for me has been KV cache pressure.
Interested in tradeoffs more than fan loyalty.
Re: context size vs speed in GGUF: worth it? - phasescope - 10-18-2026
I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Llama and compare from there.
Re: context size vs speed in GGUF: worth it? - visionyard23 - 10-18-2026
My first guess would be this: On B200, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.
Re: context size vs speed in GGUF: worth it? - pixelcrate82 - 10-18-2026
Interesting, I hadn't tried that. The pattern around workstation thermals has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: context size vs speed in GGUF: worth it? - northcraft90 - 10-18-2026
Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.
Re: context size vs speed in GGUF: worth it? - latentcraft - 10-19-2026
Quote:That matches what I was seeing. I ended up in about the same place using Ollama.
Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.
I'm happy to test one more variant if you want a second data point.
Re: context size vs speed in GGUF: worth it? - vramridge - 10-19-2026
Appreciate the detail. I don't think your conclusion is far off.
Re: context size vs speed in GGUF: worth it? - vrambandit - 10-19-2026
Good point. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: context size vs speed in GGUF: worth it? - phasescope - 10-19-2026
Interesting, I hadn't tried that. That matches what I saw on 7900 XTX.
Re: context size vs speed in GGUF: worth it? - vectorport - 10-19-2026
My first guess would be this: On B200, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.