+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: Other Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=39)
+--- Thread: throughput per watt reality check (/showthread.php?tid=104)
Pages:
1
2
throughput per watt reality check - bytehall - 09-02-2026
Curious what people here are seeing.
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling speculative decoding.
Would appreciate practical advice rather than theory.
Re: throughput per watt reality check - localpatch78 - 09-02-2026
Thanks, that helps. I ended up in about the same place using PyTorch. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: throughput per watt reality check - amberplane - 09-03-2026
Good point. I don't think your conclusion is far off.
Re: throughput per watt reality check - localpatch - 09-05-2026
I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at MoE serving. Not glamorous advice, but repeatable baselines save a lot of time.
Re: throughput per watt reality check - fastfield4 - 09-05-2026
Can you fill in a bit more context? Is this local-only or are you routing anything externally?
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.
Re: throughput per watt reality check - syshouse - 09-08-2026
that fixed it for me too. At this point the real bottleneck is my patience.
Re: throughput per watt reality check - phaseshed - 09-08-2026
A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with FLUX and compare from there.
Re: throughput per watt reality check - fastscope - 09-08-2026
Quote:A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with FLUX and compare from there.
following this because I'm in the same boat.
Re: throughput per watt reality check - oxidestream - 09-08-2026
I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.
Re: throughput per watt reality check - cozykite - 09-08-2026
Tried to reproduce it just now. On 7900 XTX with vLLM, I saw basically the same behavior. I expected the model choice to matter more than it did.
I'm happy to test one more variant if you want a second data point.