+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: Other Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=39)
+--- Thread: NVFP4 / INT4 tradeoffs: setup advice (/showthread.php?tid=548)
Pages:
1
2
NVFP4 / INT4 tradeoffs: setup advice - vramarch - 09-28-2026
Posting this in case it saves someone else a little time.
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about reranking.
Simple and boring is a valid answer.
Re: NVFP4 / INT4 tradeoffs: setup advice - flintriver50 - 09-29-2026
Ran a quick check on my box. On 5090 with Ollama, I saw basically the same behavior.
Re: NVFP4 / INT4 tradeoffs: setup advice - papermixer4 - 09-29-2026
Quote:Ran a quick check on my box. On 5090 with Ollama, I saw basically the same behavior.
I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at quant drift.
Re: NVFP4 / INT4 tradeoffs: setup advice - routecraft - 09-30-2026
A couple questions before I guess too hard. Is this local-only or are you routing anything externally? Turns out 'just one more test' was a lie.
Re: NVFP4 / INT4 tradeoffs: setup advice - southpatch - 09-30-2026
Need a little more detail here. Are you testing this on GB200 or something else?
Re: NVFP4 / INT4 tradeoffs: setup advice - vramridge - 10-02-2026
I'd start with the simpler stack.
Re: NVFP4 / INT4 tradeoffs: setup advice - batchsize - 10-02-2026
that fixed it for me too.
Re: NVFP4 / INT4 tradeoffs: setup advice - labnest - 10-02-2026
Quote:that fixed it for me too.
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at token cost. Not glamorous advice, but repeatable baselines save a lot of time.
Re: NVFP4 / INT4 tradeoffs: setup advice - cudascope - 10-05-2026
I tested this tonight. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.
Re: NVFP4 / INT4 tradeoffs: setup advice - phaseshed - 10-05-2026
Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher.