+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: Other Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=39)
+--- Thread: NVFP4 / INT4 tradeoffs for Mistral (/showthread.php?tid=559)
Pages:
1
2
NVFP4 / INT4 tradeoffs for Mistral - mangotinker73 - 09-29-2026
Posting this in case it saves someone else a little time.
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?
Would rather learn from your scars than collect my own all at once.
My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.
Re: NVFP4 / INT4 tradeoffs for Mistral - mintstream9 - 09-30-2026
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefill/decode split. Once you know the plain path works, then add the faster path back in piece by piece.
Re: NVFP4 / INT4 tradeoffs for Mistral - rocmstack25 - 09-30-2026
I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Qwen and compare from there.
Re: NVFP4 / INT4 tradeoffs for Mistral - signaltrack44 - 09-30-2026
What worked for me was pretty simple. On dual 3090, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: NVFP4 / INT4 tradeoffs for Mistral - thetashed24 - 09-30-2026
My first guess would be this: If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.
Re: NVFP4 / INT4 tradeoffs for Mistral - vectorport - 10-01-2026
Thanks, that helps. I ended up in about the same place using PyTorch. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: NVFP4 / INT4 tradeoffs for Mistral - cudapoint - 10-01-2026
Quote:Thanks, that helps. I ended up in about the same place using PyTorch. I got better results once I stopped chasing one metric and just looked at whethe
Need a little more detail here. Is this local-only or are you routing anything externally?
Re: NVFP4 / INT4 tradeoffs for Mistral - mintstream9 - 10-03-2026
Tried to reproduce it just now. On 5090 with vLLM, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.
Re: NVFP4 / INT4 tradeoffs for Mistral - orbitgarden - 10-03-2026
following this because I'm in the same boat.
Re: NVFP4 / INT4 tradeoffs for Mistral - thetatinker77 - 10-03-2026
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at KV cache pressure.