best local runtime for mixed models

[vectorhouse71]

What exact setup are you running? Is this local-only or are you routing anything externally?

[mellowwire98]

Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal.

[batchscope72]

I'd start with the simpler stack.

[vectorhouse71]

Thanks, that helps. The pattern around INT4 has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[dockridge]

Good point. That matches what I saw on dual 3090.

[edgeriver]

Interesting, I hadn't tried that. The pattern around batch size has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.