+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Hardware (https://www.datasculpting.ai/inference/forumdisplay.php?fid=4)
+--- Forum: Systems & Builds (https://www.datasculpting.ai/inference/forumdisplay.php?fid=29)
+--- Thread: power delivery for 24/7 serving (/showthread.php?tid=56)
Pages:
1
2
power delivery for 24/7 serving - edgequill - 08-29-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
Just sharing a setup that ended up working reasonably well: Gemma, TensorRT-LLM, A6000, with a lot of attention on tool calling.
Maybe useful as a starting point for newer folks.
Re: power delivery for 24/7 serving - phasenode - 08-31-2026
I'd start with the simpler stack.
Re: power delivery for 24/7 serving - latentpeak22 - 08-31-2026
same here on my side.
Re: power delivery for 24/7 serving - plainscope50 - 09-03-2026
Appreciate the detail. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
I'm happy to test one more variant if you want a second data point.
Re: power delivery for 24/7 serving - vectorcrate98 - 09-03-2026
I'd start with the simpler stack.
Re: power delivery for 24/7 serving - batchwire - 09-03-2026
My first guess would be this: If you're on Ollama, I'd confirm the version first and then reduce variables until the behavior is obvious.
Re: power delivery for 24/7 serving - sanevault - 09-04-2026
I'd approach it in a boring order. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.
Re: power delivery for 24/7 serving - tensorport - 09-04-2026
Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher.
Re: power delivery for 24/7 serving - routecraft - 09-04-2026
Need a little more detail here. Is this local-only or are you routing anything externally?
Re: power delivery for 24/7 serving - batchwire - 09-05-2026
My first guess would be this: If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious.