throughput per watt reality check

[vectorshed43]

Tried to reproduce it just now. On 7900 XTX with vLLM, I saw basically the same behavior. I expected the model choice to matter more than it did.

I tested this tonight. On dual 3090 with MLX, I saw basically the same behavior. The main thing that moved was latency, not quality.

[sysstation69]

Appreciate the detail. That matches what I saw on Threadripper box. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

If you post your exact settings, people here can probably narrow it down pretty fast.

[turbogrid11]

Appreciate the detail. The pattern around KV cache pressure has been pretty similar for me.

[dockworks23]

Not sure I agree there. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[phasespark]

Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Could be a workload difference, but I wouldn't generalize too fast from one setup.