+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: Other Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=39)
+--- Thread: NVFP4 / INT4 tradeoffs (/showthread.php?tid=1497)
NVFP4 / INT4 tradeoffs - oxidegrove40 - 10-18-2026
This keeps coming up in conversation so I figured I'd start a thread.
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with speculative decoding.
Curious whether that lines up with what other people are seeing.
This is mostly for local work rather than public serving.
Re: NVFP4 / INT4 tradeoffs - mintforge - 10-18-2026
A couple things I'd check first. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious.
Re: NVFP4 / INT4 tradeoffs - thetapath - 10-18-2026
That matches what I was seeing. The pattern around router policy has been pretty similar for me.
Re: NVFP4 / INT4 tradeoffs - modbench - 10-18-2026
Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher. I expected the model choice to matter more than it did.
Re: NVFP4 / INT4 tradeoffs - llmtrack - 10-18-2026
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tool calling. Not glamorous advice, but repeatable baselines save a lot of time.
Re: NVFP4 / INT4 tradeoffs - thetagrid15 - 10-18-2026
Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.
Re: NVFP4 / INT4 tradeoffs - kernelghost - 10-18-2026
Appreciate the detail. The pattern around INT4 has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: NVFP4 / INT4 tradeoffs - llmarch83 - 10-18-2026
A couple things I'd check first. On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.
Re: NVFP4 / INT4 tradeoffs - batchshed - 10-19-2026
What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there.
Re: NVFP4 / INT4 tradeoffs - batchyard - 10-19-2026
Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.