why this quant got weird repetition

[vramport]

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like tokens/sec decides everything downstream.

Curious whether that lines up with what other people are seeing.

[stackpilot]

Thanks, that helps. That matches what I saw on GB200. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[benchplease]

Appreciate the detail. I ended up in about the same place using LM Studio.

[southtinker]

Can you fill in a bit more context? What model, what runtime, and roughly what context size?

[northquill]

same here on my side.

[benchplease]

same here on my side.

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Mixtral and compare from there.

[cacheworks]

A couple questions before I guess too hard. Are you testing this on GB200 or something else?

[batchfield55]

Ran a quick check on my box. On Threadripper box with llama.cpp, I saw basically the same behavior. The main thing that moved was latency, not quality.