NVFP4 / INT4 tradeoffs for Llama

[localpatch78]

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: Ollama on B200, keeping MoE serving in mind from the start.

Feel free to tear it apart if you see something dumb.

I care more about stability than squeezing the last possible tokens/sec out of it.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.

[thetanest]

Thought I'd post a small recipe because a lot of threads ask similar things.

Good point. The pattern around tail latency has been pretty similar for me.

[fastfield4]

Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.

If you post your exact settings, people here can probably narrow it down pretty fast.