context size vs speed in GGUF with vLLM

[mangotinker73]

Interesting, I hadn't tried that. I ended up in about the same place using PyTorch. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

I'm happy to test one more variant if you want a second data point.