NVFP4 / INT4 tradeoffs: setup advice

[vramarch]

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about reranking.

Simple and boring is a valid answer.

[flintriver50]

Ran a quick check on my box. On 5090 with Ollama, I saw basically the same behavior.

[papermixer4]

Ran a quick check on my box. On 5090 with Ollama, I saw basically the same behavior.

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at quant drift.

[routecraft]

A couple questions before I guess too hard. Is this local-only or are you routing anything externally? Turns out 'just one more test' was a lie.

[southpatch]

Need a little more detail here. Are you testing this on GB200 or something else?

[vramridge]

I'd start with the simpler stack.

[batchsize]

that fixed it for me too.

[labnest]

that fixed it for me too.

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at token cost. Not glamorous advice, but repeatable baselines save a lot of time.

[cudascope]

I tested this tonight. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.

[phaseshed]

Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher.