+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: llama.cpp / GGUF (https://www.datasculpting.ai/inference/forumdisplay.php?fid=35)
+--- Thread: context size vs speed in GGUF: anyone testing this? (/showthread.php?tid=315)
Pages:
1
2
Re: context size vs speed in GGUF: anyone testing this? - latentcraft - 09-24-2026
Quote:I'd approach it in a boring order. On B200, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once yo
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tail latency. Not glamorous advice, but repeatable baselines save a lot of time.
Re: context size vs speed in GGUF: anyone testing this? - modbench - 09-25-2026
That matches what I was seeing. The pattern around HBM constraints has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
I'm happy to test one more variant if you want a second data point.
Re: context size vs speed in GGUF: anyone testing this? - bytesupply76 - 09-26-2026
That matches what I was seeing. I ended up in about the same place using PyTorch. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.