+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: Other Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=39)
+--- Thread: throughput per watt reality check (/showthread.php?tid=1058)
Pages:
1
2
throughput per watt reality check - mintquill - 10-12-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
Maybe useful as a starting point: vLLM on 4090, keeping MoE serving in mind from the start.
If someone wants the exact settings I can clean them up and post them.
Re: throughput per watt reality check - prompttinker44 - 10-13-2026
Quote:Thought I'd post a small recipe because a lot of threads ask similar things.
A couple things I'd check first. If you're on Ollama, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.
Re: throughput per watt reality check - mangopeak72 - 10-14-2026
A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with FLUX and compare from there.
Re: throughput per watt reality check - sysstation69 - 10-14-2026
Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.
Re: throughput per watt reality check - sparrowcraft - 10-14-2026
Interesting, I hadn't tried that. I don't think your conclusion is far off.
Re: throughput per watt reality check - bytesupply76 - 10-15-2026
Need a little more detail here. What model, what runtime, and roughly what context size?
Re: throughput per watt reality check - torchsmith25 - 10-16-2026
My first guess would be this: If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious. The logs were somehow calmer than I was.
Re: throughput per watt reality check - llmpeak - 10-18-2026
Quote:My first guess would be this: If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious. The logs were
Can you fill in a bit more context? What model, what runtime, and roughly what context size?
Re: throughput per watt reality check - benchplease - 10-18-2026
Quote:A couple things I'd check first. On 7900 XTX, the easiest win is often to back off the aggressive settings and see whether the problem survives.
A couple things I'd check first. On M3 Max, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Re: throughput per watt reality check - forgemixer - 10-18-2026
That matches what I was seeing. The pattern around continuous batching has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.