+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Optimization & Performance (https://www.datasculpting.ai/inference/forumdisplay.php?fid=10)
+--- Forum: Performance Tuning (https://www.datasculpting.ai/inference/forumdisplay.php?fid=42)
+--- Thread: consumer GPU serving economics (/showthread.php?tid=1029)
consumer GPU serving economics - sparrowshed - 10-10-2026
Did a bit of testing and thought I'd post the result here.
Short version: Ollama behaved well for me, but only after I stopped ignoring embeddings.
Nothing scientific here, just practical notes from a real setup.
Re: consumer GPU serving economics - cudapeak - 10-11-2026
A couple things I'd check first. On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Re: consumer GPU serving economics - southtinker - 10-14-2026
That matches what I was seeing. I ended up in about the same place using MLX. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: consumer GPU serving economics - batchember - 10-15-2026
A couple things I'd check first. If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious.
Re: consumer GPU serving economics - bytestack - 10-17-2026
Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly.
Re: consumer GPU serving economics - bytelab - 10-17-2026
Quote:Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly.
subscribed, curious where this lands.
Re: consumer GPU serving economics - kernelrunner - 10-17-2026
following this because I'm in the same boat.
I'm happy to test one more variant if you want a second data point.
Re: consumer GPU serving economics - rocmhouse90 - 10-17-2026
I'd start with the simpler stack.
Re: consumer GPU serving economics - batchember - 10-17-2026
Not sure I agree there. I've seen people blame KV cache pressure when the real issue was much more mundane.