4090 vs 5090 for 70B quants on dual 3090

[paperrunner]

I tested this tonight. My result was mixed: first run looked bad, second run looked normal.

What exact setup are you running? Is this local-only or are you routing anything externally?

[rocmriley]

That matches what I was seeing. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[signaltrack44]

Interesting, I hadn't tried that. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[cozysmith]

I'd approach it in a boring order. On M3 Max, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[amberhall]

My first guess would be this: On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[signaltrack44]

that fixed it for me too.