quant choices for 70B local with PyTorch

[fusedtinker]

Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. I expected the model choice to

That matches what I was seeing. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

I'm happy to test one more variant if you want a second data point.