4090 vs 5090 for 70B quants for Mixtral

[vrambandit]

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: DeepSeek, TensorRT-LLM, 7900 XTX, with a lot of attention on KV cache pressure.

Feel free to tear it apart if you see something dumb.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

[amberhall]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Qwen and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[thetarunner]

Thanks, that helps. The pattern around VRAM headroom has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[circuitwire58]

Good point. The pattern around HBM constraints has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[vrambandit]

A couple things I'd check first. On 4090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

[signaltrack44]

Need a little more detail here. What model, what runtime, and roughly what context size?

[dockkite69]

I think the bottleneck is somewhere else. I've seen people blame speculative decoding when the real issue was much more mundane. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[rocmstack25]

Good point. That matches what I saw on Threadripper box.

[kernelmike]

Good point. That matches what I saw on Threadripper box.

A couple things I'd check first. On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

[fusedhall]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Llama and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.