+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: Other Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=39)
+--- Thread: NVFP4 / INT4 tradeoffs (/showthread.php?tid=526)
Pages:
1
2
NVFP4 / INT4 tradeoffs - northhall88 - 09-28-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
Maybe useful as a starting point: LM Studio on M3 Max, keeping structured output in mind from the start.
If someone wants the exact settings I can clean them up and post them.
Re: NVFP4 / INT4 tradeoffs - stackarch - 09-29-2026
same here on my side.
Re: NVFP4 / INT4 tradeoffs - visionforge - 09-29-2026
My first guess would be this: On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.
Re: NVFP4 / INT4 tradeoffs - cachebench - 10-02-2026
Maybe, but I had the opposite result. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the simpler explanation before adopting the more interesting one.
Re: NVFP4 / INT4 tradeoffs - turboscope - 10-03-2026
Quote:Maybe, but I had the opposite result. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Wort
Need a little more detail here. Are you testing this on B200 or something else?
Re: NVFP4 / INT4 tradeoffs - quietcraft - 10-06-2026
I tested this tonight. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.
Re: NVFP4 / INT4 tradeoffs - audiodesk - 10-09-2026
Appreciate the detail. That matches what I saw on 7900 XTX. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: NVFP4 / INT4 tradeoffs - relaysupply90 - 10-10-2026
Quote:Appreciate the detail. That matches what I saw on 7900 XTX. Worth rechecking your defaults though, because one bad assumption can make the rest of the
Can you fill in a bit more context? What model, what runtime, and roughly what context size?
Re: NVFP4 / INT4 tradeoffs - kernelhouse70 - 10-10-2026
that fixed it for me too.
Re: NVFP4 / INT4 tradeoffs - rocmcrate26 - 10-10-2026
What exact setup are you running? Are you testing this on Threadripper box or something else?