Inference
4090 vs 5090 for 70B quants for Mixtral - Printable Version

+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Hardware (https://www.datasculpting.ai/inference/forumdisplay.php?fid=4)
+--- Forum: GPUs (https://www.datasculpting.ai/inference/forumdisplay.php?fid=28)
+--- Thread: 4090 vs 5090 for 70B quants for Mixtral (/showthread.php?tid=1133)

Pages: 1 2

4090 vs 5090 for 70B quants for Mixtral - vrambandit - 10-12-2026

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: DeepSeek, TensorRT-LLM, 7900 XTX, with a lot of attention on KV cache pressure.

Feel free to tear it apart if you see something dumb.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

Re: 4090 vs 5090 for 70B quants for Mixtral - amberhall - 10-12-2026

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Qwen and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

Re: 4090 vs 5090 for 70B quants for Mixtral - thetarunner - 10-12-2026

Thanks, that helps. The pattern around VRAM headroom has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

Re: 4090 vs 5090 for 70B quants for Mixtral - circuitwire58 - 10-12-2026

Good point. The pattern around HBM constraints has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

Re: 4090 vs 5090 for 70B quants for Mixtral - vrambandit - 10-12-2026

A couple things I'd check first. On 4090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

Re: 4090 vs 5090 for 70B quants for Mixtral - signaltrack44 - 10-13-2026

Need a little more detail here. What model, what runtime, and roughly what context size?

Re: 4090 vs 5090 for 70B quants for Mixtral - dockkite69 - 10-13-2026

I think the bottleneck is somewhere else. I've seen people blame speculative decoding when the real issue was much more mundane. Could be a workload difference, but I wouldn't generalize too fast from one setup.

Re: 4090 vs 5090 for 70B quants for Mixtral - rocmstack25 - 10-13-2026

Good point. That matches what I saw on Threadripper box.

Re: 4090 vs 5090 for 70B quants for Mixtral - kernelmike - 10-13-2026

Quote:Good point. That matches what I saw on Threadripper box.

A couple things I'd check first. On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

Re: 4090 vs 5090 for 70B quants for Mixtral - fusedhall - 10-13-2026

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Llama and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.