+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Optimization & Performance (https://www.datasculpting.ai/inference/forumdisplay.php?fid=10)
+--- Forum: Quantization (https://www.datasculpting.ai/inference/forumdisplay.php?fid=40)
+--- Thread: why this quant got weird repetition (/showthread.php?tid=68)
why this quant got weird repetition - vramport - 08-30-2026
This keeps coming up in conversation so I figured I'd start a thread.
The more I work with local setups, the more it seems like tokens/sec decides everything downstream.
Curious whether that lines up with what other people are seeing.
Re: why this quant got weird repetition - stackpilot - 08-30-2026
Thanks, that helps. That matches what I saw on GB200. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: why this quant got weird repetition - benchplease - 08-30-2026
Appreciate the detail. I ended up in about the same place using LM Studio.
Re: why this quant got weird repetition - southtinker - 08-30-2026
Can you fill in a bit more context? What model, what runtime, and roughly what context size?
Re: why this quant got weird repetition - northquill - 08-30-2026
same here on my side.
Re: why this quant got weird repetition - benchplease - 08-30-2026
Quote:same here on my side.
What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Mixtral and compare from there.
Re: why this quant got weird repetition - cacheworks - 09-02-2026
A couple questions before I guess too hard. Are you testing this on GB200 or something else?
Re: why this quant got weird repetition - batchfield55 - 09-02-2026
Ran a quick check on my box. On Threadripper box with llama.cpp, I saw basically the same behavior. The main thing that moved was latency, not quality.