consumer GPU serving economics

[macropilot]

A couple things I'd check first. If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of th

worth checking the driver first.

[northquill]

A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?

[northhall88]

A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?

I'd approach it in a boring order. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[bytestack]

Appreciate the detail. I ended up in about the same place using llama.cpp.