+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Optimization & Performance (https://www.datasculpting.ai/inference/forumdisplay.php?fid=10)
+--- Forum: Quantization (https://www.datasculpting.ai/inference/forumdisplay.php?fid=40)
+--- Thread: quant choices for 70B local: what are you seeing? (/showthread.php?tid=258)
quant choices for 70B local: what are you seeing? - vectorport - 09-13-2026
Spent some time on this today and wanted to compare notes.
I've gone back and forth between PyTorch and vLLM for DeepSeek. The interesting difference for me has been VRAM headroom.
Interested in tradeoffs more than fan loyalty.
I care more about stability than squeezing the last possible tokens/sec out of it.
Re: quant choices for 70B local: what are you seeing? - relaynest82 - 09-13-2026
I hit almost the same issue last night.
Re: quant choices for 70B local: what are you seeing? - llmarch83 - 09-14-2026
A couple things I'd check first. If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.
I'm happy to test one more variant if you want a second data point.
Re: quant choices for 70B local: what are you seeing? - southtinker - 09-15-2026
same here on my side.
Re: quant choices for 70B local: what are you seeing? - northquill - 09-15-2026
What exact setup are you running? Are you testing this on Threadripper box or something else? The logs were somehow calmer than I was.
Re: quant choices for 70B local: what are you seeing? - benchplease - 09-15-2026
What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tool calling.
Re: quant choices for 70B local: what are you seeing? - bytehall - 09-17-2026
My first guess would be this: If you're on MLX, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: quant choices for 70B local: what are you seeing? - llmarch51 - 09-17-2026
Ran a quick check on my box. On Threadripper box with LM Studio, I saw basically the same behavior.
Re: quant choices for 70B local: what are you seeing? - vrambandit - 09-17-2026
following this because I'm in the same boat.
Re: quant choices for 70B local: what are you seeing? - fusedtinker - 09-18-2026
Need a little more detail here. Do you see the same thing with a smaller batch or shorter prompt?
I'm happy to test one more variant if you want a second data point.