NVFP4 / INT4 tradeoffs

[llmarch83]

A couple things I'd check first. On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.