throughput per watt reality check

[mintquill]

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: vLLM on 4090, keeping MoE serving in mind from the start.

If someone wants the exact settings I can clean them up and post them.

[prompttinker44]

Thought I'd post a small recipe because a lot of threads ask similar things.

A couple things I'd check first. If you're on Ollama, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.

[mangopeak72]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with FLUX and compare from there.

[sysstation69]

Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.

[sparrowcraft]

Interesting, I hadn't tried that. I don't think your conclusion is far off.

[bytesupply76]

Need a little more detail here. What model, what runtime, and roughly what context size?

[torchsmith25]

My first guess would be this: If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious. The logs were somehow calmer than I was.

[llmpeak]

My first guess would be this: If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious. The logs were

Can you fill in a bit more context? What model, what runtime, and roughly what context size?

[benchplease]

A couple things I'd check first. On 7900 XTX, the easiest win is often to back off the aggressive settings and see whether the problem survives.

A couple things I'd check first. On M3 Max, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[forgemixer]

That matches what I was seeing. The pattern around continuous batching has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.