quant choices for 70B local on GB200

[cudaline]

Curious what people here are seeing.

I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling reranking.

If you've tested something similar, what did you keep and what did you stop bothering with?

[cobalttinker]

Curious what people here are seeing.

Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.

[vrambandit]

I'd approach it in a boring order. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[paperrunner]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Kimi and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[fusedtinker]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Kimi and compare from there. A lot of these threads en

that fixed it for me too.

If you post your exact settings, people here can probably narrow it down pretty fast.

[benchplease]

[quote]I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Kimi and compare from there. A lot of these thr

I tested this tonight. On B200 with PyTorch, I saw basically the same behavior.