+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Optimization & Performance (https://www.datasculpting.ai/inference/forumdisplay.php?fid=10)
+--- Forum: Performance Tuning (https://www.datasculpting.ai/inference/forumdisplay.php?fid=42)
+--- Thread: consumer GPU serving economics (/showthread.php?tid=1342)
Pages:
1
2
consumer GPU serving economics - kernelrunner - 10-16-2026
Posting this in case it saves someone else a little time.
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about reranking.
I'm especially interested in mistakes worth avoiding.
I care more about stability than squeezing the last possible tokens/sec out of it.
I expected the biggest difference to come from the model, but the stack around it mattered at least as much.
Re: consumer GPU serving economics - tensorfield88 - 10-16-2026
A couple things I'd check first. On dual 3090, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Re: consumer GPU serving economics - kernelrunner - 10-16-2026
What exact setup are you running? What model, what runtime, and roughly what context size?
Re: consumer GPU serving economics - localpatch78 - 10-16-2026
Can you fill in a bit more context? What model, what runtime, and roughly what context size?
I'm happy to test one more variant if you want a second data point.
Re: consumer GPU serving economics - papermixer4 - 10-16-2026
Quote:Can you fill in a bit more context? What model, what runtime, and roughly what context size?
that fixed it for me too.
Re: consumer GPU serving economics - sparrowshed - 10-17-2026
Interesting, I hadn't tried that. That matches what I saw on GB200.
Re: consumer GPU serving economics - kernelmike - 10-18-2026
Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.
Re: consumer GPU serving economics - marbletrack - 10-18-2026
Quote:Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more
Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.
Re: consumer GPU serving economics - signalcraft14 - 10-18-2026
Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. I expected the model choice to matter more than it did.
Re: consumer GPU serving economics - kernelmike - 10-18-2026
Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.