context size vs speed in GGUF with vLLM

[embershed99]

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of PyTorch and B200 felt surprisingly usable once I tuned around consumer GPUs.

Feel free to tear it apart if you see something dumb.

[vrambandit]

Good point. That matches what I saw on 4090. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[vectorport]

I hit almost the same issue last night.

[audioriver]

Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.

[mangotinker73]

I'd start with the simpler stack.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[mellowwire98]

Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal.

[pixelcrate82]

I tested this tonight. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.

[cudacrate]

I'd start with the simpler stack.

Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher. I expected the model choice to matter more than it did.

[mangotinker73]

Interesting, I hadn't tried that. I ended up in about the same place using PyTorch. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

I'm happy to test one more variant if you want a second data point.

[latentcraft]

A couple things I'd check first. On 4090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

I'm happy to test one more variant if you want a second data point.