+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Optimization & Performance (https://www.datasculpting.ai/inference/forumdisplay.php?fid=10)
+--- Forum: Quantization (https://www.datasculpting.ai/inference/forumdisplay.php?fid=40)
+--- Thread: AWQ vs GPTQ vs GGUF right now on mini PC (/showthread.php?tid=1084)
Pages:
1
2
AWQ vs GPTQ vs GGUF right now on mini PC - cacheworks - 10-11-2026
Posting this in case it saves someone else a little time.
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?
I'm especially interested in mistakes worth avoiding.
My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.
Re: AWQ vs GPTQ vs GGUF right now on mini PC - llmarch51 - 10-12-2026
Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal.
Re: AWQ vs GPTQ vs GGUF right now on mini PC - torchshed - 10-13-2026
What exact setup are you running? What model, what runtime, and roughly what context size?
Re: AWQ vs GPTQ vs GGUF right now on mini PC - phasestream53 - 10-13-2026
Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.
Re: AWQ vs GPTQ vs GGUF right now on mini PC - cacheworks - 10-14-2026
Interesting, I hadn't tried that. The pattern around batch size has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: AWQ vs GPTQ vs GGUF right now on mini PC - llmarch51 - 10-14-2026
Appreciate the detail. I don't think your conclusion is far off.
Re: AWQ vs GPTQ vs GGUF right now on mini PC - bytehall - 10-15-2026
Thanks, that helps. The pattern around VRAM headroom has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: AWQ vs GPTQ vs GGUF right now on mini PC - turboscope - 10-16-2026
Quote:Thanks, that helps. The pattern around VRAM headroom has been pretty similar for me. The biggest change for me was simplifying the stack instead of ad
Appreciate the detail. The pattern around HBM constraints has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: AWQ vs GPTQ vs GGUF right now on mini PC - mangofield33 - 10-16-2026
A couple questions before I guess too hard. Are you testing this on M3 Max or something else?
Re: AWQ vs GPTQ vs GGUF right now on mini PC - southtinker - 10-18-2026
I think the bottleneck is somewhere else. I get why that seems right, but on mini PC the bottleneck felt elsewhere. Worth testing the simpler explanation before adopting the more interesting one.