+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Hardware (https://www.datasculpting.ai/inference/forumdisplay.php?fid=4)
+--- Forum: Systems & Builds (https://www.datasculpting.ai/inference/forumdisplay.php?fid=29)
+--- Thread: power delivery for 24/7 serving (/showthread.php?tid=112)
Pages:
1
2
power delivery for 24/7 serving - sanescope - 09-03-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
Maybe useful as a starting point: MLX on GB200, keeping prefill/decode split in mind from the start.
Feel free to tear it apart if you see something dumb.
Re: power delivery for 24/7 serving - tensorworks - 09-03-2026
I'd approach it in a boring order. If you're on Ollama, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: power delivery for 24/7 serving - batchpeak - 09-03-2026
I hit almost the same issue last night.
Re: power delivery for 24/7 serving - batchwire - 09-03-2026
Quote:I hit almost the same issue last night.
Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly.
Re: power delivery for 24/7 serving - tensorworks - 09-03-2026
Can you fill in a bit more context? Is this local-only or are you routing anything externally?
Re: power delivery for 24/7 serving - stackpilot - 09-03-2026
Quote:Can you fill in a bit more context? Is this local-only or are you routing anything externally?
I'd start with the simpler stack.
Re: power delivery for 24/7 serving - bytegarden73 - 09-04-2026
Quote:Quote:Can you fill in a bit more context? Is this local-only or are you routing anything externally?
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tail latency. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: power delivery for 24/7 serving - batchsize - 09-04-2026
Not sure I agree there. I get why that seems right, but on dual 3090 the bottleneck felt elsewhere. Could be a workload difference, but I wouldn't generalize too fast from one setup.
Re: power delivery for 24/7 serving - stackpilot - 09-05-2026
Ran a quick check on my box. On Threadripper box with vLLM, I saw basically the same behavior. I expected the model choice to matter more than it did.
Re: power delivery for 24/7 serving - edgequill - 09-06-2026
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at token cost. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.