context size vs speed in GGUF: worth it?

[visionyard23]

Spent some time on this today and wanted to compare notes.

I've gone back and forth between Ollama and TensorRT-LLM for Mistral. The interesting difference for me has been KV cache pressure.

Interested in tradeoffs more than fan loyalty.

[phasescope]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Llama and compare from there.

[visionyard23]

My first guess would be this: On B200, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

[pixelcrate82]

Interesting, I hadn't tried that. The pattern around workstation thermals has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[northcraft90]

Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.

[latentcraft]

That matches what I was seeing. I ended up in about the same place using Ollama.

Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.

I'm happy to test one more variant if you want a second data point.

[vramridge]

Appreciate the detail. I don't think your conclusion is far off.

[vrambandit]

Good point. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[phasescope]

Interesting, I hadn't tried that. That matches what I saw on 7900 XTX.

[vectorport]

My first guess would be this: On B200, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.