consumer GPU serving economics

[localpatch78]

[quote][quote]Not sure I agree there. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Not saying you're

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tail latency. Once you know the plain path works, then add the faster path back in piece by piece.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.