+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Optimization & Performance (https://www.datasculpting.ai/inference/forumdisplay.php?fid=10)
+--- Forum: Performance Tuning (https://www.datasculpting.ai/inference/forumdisplay.php?fid=42)
+--- Thread: consumer GPU serving economics on 7900 XTX (/showthread.php?tid=710)
Pages:
1
2
consumer GPU serving economics on 7900 XTX - kernelrunner - 10-04-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
Just sharing a setup that ended up working reasonably well: Mistral, PyTorch, A6000, with a lot of attention on consumer GPUs.
Maybe useful as a starting point for newer folks.
I care more about stability than squeezing the last possible tokens/sec out of it.
Re: consumer GPU serving economics on 7900 XTX - kernelrunner - 10-04-2026
I think the bottleneck is somewhere else. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Worth testing the simpler explanation before adopting the more interesting one.
Re: consumer GPU serving economics on 7900 XTX - vramport - 10-07-2026
My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Phi and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.
Re: consumer GPU serving economics on 7900 XTX - sparrowshed - 10-07-2026
Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher.
Re: consumer GPU serving economics on 7900 XTX - kernelrunner - 10-07-2026
Quote:Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher.
I'd start with the simpler stack.
Re: consumer GPU serving economics on 7900 XTX - rocmpath - 10-07-2026
I tested this tonight. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.
Re: consumer GPU serving economics on 7900 XTX - stackarch - 10-07-2026
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tool calling. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it. Turns out 'just one more test' was a lie.
Re: consumer GPU serving economics on 7900 XTX - fusedhall56 - 10-07-2026
My first guess would be this: On B200, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.
Re: consumer GPU serving economics on 7900 XTX - prompttinker44 - 10-07-2026
Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.
Re: consumer GPU serving economics on 7900 XTX - localpatch78 - 10-07-2026
that fixed it for me too.