+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: Other Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=39)
+--- Thread: NVFP4 / INT4 tradeoffs for Llama (/showthread.php?tid=575)
NVFP4 / INT4 tradeoffs for Llama - localpatch78 - 09-29-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
Maybe useful as a starting point: Ollama on B200, keeping MoE serving in mind from the start.
Feel free to tear it apart if you see something dumb.
I care more about stability than squeezing the last possible tokens/sec out of it.
If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.
Re: NVFP4 / INT4 tradeoffs for Llama - thetanest - 09-29-2026
Quote:Thought I'd post a small recipe because a lot of threads ask similar things.
Good point. The pattern around tail latency has been pretty similar for me.
Re: NVFP4 / INT4 tradeoffs for Llama - fastfield4 - 09-29-2026
Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.
If you post your exact settings, people here can probably narrow it down pretty fast.