AWQ vs GPTQ vs GGUF right now on A6000

[cudahouse]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Mixtral and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.

[benchplease]

Appreciate the detail. I don't think your conclusion is far off.

[marbletrack]

My first guess would be this: If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.

If you post your exact settings, people here can probably narrow it down pretty fast.

[cudaline]

My first guess would be this: If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Not gl

Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal.

[lablab]

Can you fill in a bit more context? What model, what runtime, and roughly what context size?

[benchplease]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with DeepSeek and compare from there.