+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Optimization & Performance (https://www.datasculpting.ai/inference/forumdisplay.php?fid=10)
+--- Forum: Quantization (https://www.datasculpting.ai/inference/forumdisplay.php?fid=40)
+--- Thread: quant choices for 70B local for Llama (/showthread.php?tid=1190)
quant choices for 70B local for Llama - llmarch51 - 10-13-2026
Has anyone else run into this?
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling VRAM headroom.
If you've tested something similar, what did you keep and what did you stop bothering with?
Re: quant choices for 70B local for Llama - vrambandit - 10-13-2026
Good point. The pattern around speculative decoding has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: quant choices for 70B local for Llama - southtinker - 10-14-2026
Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the simpler explanation before adopting the more interesting one.
Re: quant choices for 70B local for Llama - northquill - 10-15-2026
Quote:Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the
My first guess would be this: If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: quant choices for 70B local for Llama - vrambandit - 10-15-2026
Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher. I expected the model choice to matter more than it did.
Re: quant choices for 70B local for Llama - quiettoken - 10-16-2026
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at NVFP4. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.