+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Optimization & Performance (https://www.datasculpting.ai/inference/forumdisplay.php?fid=10)
+--- Forum: Quantization (https://www.datasculpting.ai/inference/forumdisplay.php?fid=40)
+--- Thread: quant choices for 70B local on GB200 (/showthread.php?tid=1136)
quant choices for 70B local on GB200 - cudaline - 10-13-2026
Curious what people here are seeing.
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling reranking.
If you've tested something similar, what did you keep and what did you stop bothering with?
Re: quant choices for 70B local on GB200 - cobalttinker - 10-13-2026
Quote:Curious what people here are seeing.
Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.
Re: quant choices for 70B local on GB200 - vrambandit - 10-13-2026
I'd approach it in a boring order. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: quant choices for 70B local on GB200 - paperrunner - 10-15-2026
I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Kimi and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: quant choices for 70B local on GB200 - fusedtinker - 10-15-2026
Quote:I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Kimi and compare from there. A lot of these threads en
that fixed it for me too.
If you post your exact settings, people here can probably narrow it down pretty fast.
Re: quant choices for 70B local on GB200 - benchplease - 10-15-2026
Quote:[quote]I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Kimi and compare from there. A lot of these thr
I tested this tonight. On B200 with PyTorch, I saw basically the same behavior.