+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Optimization & Performance (https://www.datasculpting.ai/inference/forumdisplay.php?fid=10)
+--- Forum: Quantization (https://www.datasculpting.ai/inference/forumdisplay.php?fid=40)
+--- Thread: quant choices for 70B local (/showthread.php?tid=435)
quant choices for 70B local - paperrunner - 09-23-2026
Spent some time on this today and wanted to compare notes.
I'm comparing two approaches here: keep it simple on Threadripper box, or squeeze more out of the stack and deal with the complexity.
Trying to avoid rebuilding the whole thing twice.
I care more about stability than squeezing the last possible tokens/sec out of it.