+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Hardware (https://www.datasculpting.ai/inference/forumdisplay.php?fid=4)
+--- Forum: Systems & Builds (https://www.datasculpting.ai/inference/forumdisplay.php?fid=29)
+--- Thread: power delivery for 24/7 serving with vLLM (/showthread.php?tid=1412)
Pages:
1
2
power delivery for 24/7 serving with vLLM - nanobrook - 10-17-2026
This keeps coming up in conversation so I figured I'd start a thread.
The more I work with local setups, the more it seems like MoE serving decides everything downstream.
Wondering if this is just where the whole space is heading now.
Re: power delivery for 24/7 serving with vLLM - batchyard - 10-18-2026
Quote:This keeps coming up in conversation so I figured I'd start a thread.
Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.
Re: power delivery for 24/7 serving with vLLM - tensorworks - 10-18-2026
worth checking the driver first.
Re: power delivery for 24/7 serving with vLLM - bytegarden73 - 10-18-2026
I don't think that's the whole story. I get why that seems right, but on Threadripper box the bottleneck felt elsewhere. Worth testing the simpler explanation before adopting the more interesting one.
Re: power delivery for 24/7 serving with vLLM - cozygrid12 - 10-18-2026
I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at small local stacks. Not glamorous advice, but repeatable baselines save a lot of time.
Re: power delivery for 24/7 serving with vLLM - bytegarden73 - 10-18-2026
My first guess would be this: On 7900 XTX, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.
Re: power delivery for 24/7 serving with vLLM - phasenode - 10-18-2026
Need a little more detail here. Is this local-only or are you routing anything externally?
Re: power delivery for 24/7 serving with vLLM - batchyard - 10-18-2026
I don't think that's the whole story. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Could be a workload difference, but I wouldn't generalize too fast from one setup.
Re: power delivery for 24/7 serving with vLLM - nanobrook - 10-18-2026
Thanks, that helps. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: power delivery for 24/7 serving with vLLM - homelabhenry - 10-18-2026
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at router policy. Not glamorous advice, but repeatable baselines save a lot of time.