why this quant got weird repetition

[rocmriley]

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about tokens/sec.

I'm especially interested in mistakes worth avoiding.

I care more about stability than squeezing the last possible tokens/sec out of it.

[northquill]

same here on my side.

[relaynest82]

same here on my side.

[cudaline]

That matches what I was seeing. I ended up in about the same place using llama.cpp.

[vrambandit]

Appreciate the detail. I ended up in about the same place using PyTorch. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[cudaworks]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Phi and compare from there. Once you know the plain path works, then add the faster path back in piece by piece. The logs were somehow calmer than I was.

[marbletrack]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Phi and compare from there. Once you know the plain pa

I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher. I expected the model choice to matter more than it did.

[mangofield33]

[quote]I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Phi and compare from there. Once you know the p

subscribed, curious where this lands.

[edgeriver]

Need a little more detail here. Are you testing this on 5090 or something else?

[bytehall]

following this because I'm in the same boat.