+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Optimization & Performance (https://www.datasculpting.ai/inference/forumdisplay.php?fid=10)
+--- Forum: Quantization (https://www.datasculpting.ai/inference/forumdisplay.php?fid=40)
+--- Thread: quant choices for 70B local: what are you seeing? (/showthread.php?tid=660)
quant choices for 70B local: what are you seeing? - southtinker - 10-01-2026
Posting this in case it saves someone else a little time.
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?
Simple and boring is a valid answer.
This is mostly for local work rather than public serving.
Re: quant choices for 70B local: what are you seeing? - mangofield33 - 10-02-2026
I'd approach it in a boring order. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
If you post your exact settings, people here can probably narrow it down pretty fast.
Re: quant choices for 70B local: what are you seeing? - benchplease - 10-02-2026
That matches what I was seeing. The pattern around small local stacks has been pretty similar for me.
Re: quant choices for 70B local: what are you seeing? - vramscope - 10-05-2026
A couple questions before I guess too hard. Is this local-only or are you routing anything externally?
Re: quant choices for 70B local: what are you seeing? - paperrunner - 10-08-2026
What exact setup are you running? Do you see the same thing with a smaller batch or shorter prompt?
Re: quant choices for 70B local: what are you seeing? - vrambandit - 10-09-2026
I don't think that's the whole story. I get why that seems right, but on GB200 the bottleneck felt elsewhere. Could be a workload difference, but I wouldn't generalize too fast from one setup.
Re: quant choices for 70B local: what are you seeing? - oxidegrove40 - 10-09-2026
worth checking the driver first.
Re: quant choices for 70B local: what are you seeing? - nanobrook - 10-12-2026
Interesting, I hadn't tried that. That matches what I saw on dual 3090. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: quant choices for 70B local: what are you seeing? - phasestream53 - 10-12-2026
Quote:Interesting, I hadn't tried that. That matches what I saw on dual 3090. Worth rechecking your defaults though, because one bad assumption can make the
My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Kimi and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.