power delivery for 24/7 serving

[sanescope]

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: MLX on GB200, keeping prefill/decode split in mind from the start.

Feel free to tear it apart if you see something dumb.

[tensorworks]

I'd approach it in a boring order. If you're on Ollama, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[batchpeak]

I hit almost the same issue last night.

[batchwire]

I hit almost the same issue last night.

Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly.

[tensorworks]

Can you fill in a bit more context? Is this local-only or are you routing anything externally?

[stackpilot]

Can you fill in a bit more context? Is this local-only or are you routing anything externally?

I'd start with the simpler stack.

[bytegarden73]

Can you fill in a bit more context? Is this local-only or are you routing anything externally?

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tail latency. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[batchsize]

Not sure I agree there. I get why that seems right, but on dual 3090 the bottleneck felt elsewhere. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[stackpilot]

Ran a quick check on my box. On Threadripper box with vLLM, I saw basically the same behavior. I expected the model choice to matter more than it did.

[edgequill]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at token cost. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.