power delivery for 24/7 serving with vLLM

[localpatch]

same here on my side.

[thetafield]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at quant drift. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[bytegarden73]

I don't think that's the whole story. I've seen people blame embeddings when the real issue was much more mundane. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[visionforge]

Need a little more detail here. Is this local-only or are you routing anything externally?

Good point. I ended up in about the same place using vLLM. The biggest change for me was simplifying the stack instead of adding one more optimization layer.