context size vs speed in GGUF: anyone testing this?

[latentcraft]

I'd approach it in a boring order. On B200, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once yo

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tail latency. Not glamorous advice, but repeatable baselines save a lot of time.

[modbench]

That matches what I was seeing. The pattern around HBM constraints has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

I'm happy to test one more variant if you want a second data point.

[bytesupply76]

That matches what I was seeing. I ended up in about the same place using PyTorch. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.