power delivery for 24/7 serving

[edgequill]

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Gemma, TensorRT-LLM, A6000, with a lot of attention on tool calling.

Maybe useful as a starting point for newer folks.

[phasenode]

I'd start with the simpler stack.

[latentpeak22]

same here on my side.

[plainscope50]

Appreciate the detail. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

I'm happy to test one more variant if you want a second data point.

[vectorcrate98]

I'd start with the simpler stack.

[batchwire]

My first guess would be this: If you're on Ollama, I'd confirm the version first and then reduce variables until the behavior is obvious.

[sanevault]

I'd approach it in a boring order. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

[tensorport]

Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher.

[routecraft]

Need a little more detail here. Is this local-only or are you routing anything externally?

[batchwire]

My first guess would be this: If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious.