throughput per watt reality check

[bytehall]

Curious what people here are seeing.

I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling speculative decoding.

Would appreciate practical advice rather than theory.

[localpatch78]

Thanks, that helps. I ended up in about the same place using PyTorch. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[amberplane]

Good point. I don't think your conclusion is far off.

[localpatch]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at MoE serving. Not glamorous advice, but repeatable baselines save a lot of time.

[fastfield4]

Can you fill in a bit more context? Is this local-only or are you routing anything externally?

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[syshouse]

that fixed it for me too. At this point the real bottleneck is my patience.

[phaseshed]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with FLUX and compare from there.

[fastscope]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with FLUX and compare from there.

following this because I'm in the same boat.

[oxidestream]

I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.

[cozykite]

Tried to reproduce it just now. On 7900 XTX with vLLM, I saw basically the same behavior. I expected the model choice to matter more than it did.

I'm happy to test one more variant if you want a second data point.