+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: Other Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=39)
+--- Thread: throughput per watt reality check (/showthread.php?tid=104)
Pages:
1
2
Re: throughput per watt reality check - vectorshed43 - 09-08-2026
Quote:Tried to reproduce it just now. On 7900 XTX with vLLM, I saw basically the same behavior. I expected the model choice to matter more than it did.
I tested this tonight. On dual 3090 with MLX, I saw basically the same behavior. The main thing that moved was latency, not quality.
Re: throughput per watt reality check - sysstation69 - 09-10-2026
Appreciate the detail. That matches what I saw on Threadripper box. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
If you post your exact settings, people here can probably narrow it down pretty fast.
Re: throughput per watt reality check - turbogrid11 - 09-11-2026
Appreciate the detail. The pattern around KV cache pressure has been pretty similar for me.
Re: throughput per watt reality check - dockworks23 - 09-11-2026
Not sure I agree there. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.
Re: throughput per watt reality check - phasespark - 09-14-2026
Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Could be a workload difference, but I wouldn't generalize too fast from one setup.