consumer GPU serving economics

[northhall88]

A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?

I'd approach it in a boring order. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survives.