AWQ vs GPTQ vs GGUF right now on mini PC

[cacheworks]

Posting this in case it saves someone else a little time.

If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

I'm especially interested in mistakes worth avoiding.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

[llmarch51]

Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal.

[torchshed]

What exact setup are you running? What model, what runtime, and roughly what context size?

[phasestream53]

Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.

[cacheworks]

Interesting, I hadn't tried that. The pattern around batch size has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[llmarch51]

Appreciate the detail. I don't think your conclusion is far off.

[bytehall]

Thanks, that helps. The pattern around VRAM headroom has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[turboscope]

Thanks, that helps. The pattern around VRAM headroom has been pretty similar for me. The biggest change for me was simplifying the stack instead of ad

Appreciate the detail. The pattern around HBM constraints has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[mangofield33]

A couple questions before I guess too hard. Are you testing this on M3 Max or something else?

[southtinker]

I think the bottleneck is somewhere else. I get why that seems right, but on mini PC the bottleneck felt elsewhere. Worth testing the simpler explanation before adopting the more interesting one.