+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Optimization & Performance (https://www.datasculpting.ai/inference/forumdisplay.php?fid=10)
+--- Forum: Quantization (https://www.datasculpting.ai/inference/forumdisplay.php?fid=40)
+--- Thread: why this quant got weird repetition (/showthread.php?tid=813)
Pages:
1
2
why this quant got weird repetition - rocmriley - 10-06-2026
Posting this in case it saves someone else a little time.
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about tokens/sec.
I'm especially interested in mistakes worth avoiding.
I care more about stability than squeezing the last possible tokens/sec out of it.
Re: why this quant got weird repetition - northquill - 10-07-2026
same here on my side.
Re: why this quant got weird repetition - relaynest82 - 10-07-2026
same here on my side.
Re: why this quant got weird repetition - cudaline - 10-07-2026
That matches what I was seeing. I ended up in about the same place using llama.cpp.
Re: why this quant got weird repetition - vrambandit - 10-08-2026
Appreciate the detail. I ended up in about the same place using PyTorch. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: why this quant got weird repetition - cudaworks - 10-10-2026
I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Phi and compare from there. Once you know the plain path works, then add the faster path back in piece by piece. The logs were somehow calmer than I was.
Re: why this quant got weird repetition - marbletrack - 10-10-2026
Quote:I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Phi and compare from there. Once you know the plain pa
I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher. I expected the model choice to matter more than it did.
Re: why this quant got weird repetition - mangofield33 - 10-10-2026
Quote:[quote]I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Phi and compare from there. Once you know the p
subscribed, curious where this lands.
Re: why this quant got weird repetition - edgeriver - 10-10-2026
Need a little more detail here. Are you testing this on 5090 or something else?
Re: why this quant got weird repetition - bytehall - 10-10-2026
following this because I'm in the same boat.