consumer GPU serving economics

[cudapeak]

Did a bit of testing and thought I'd post the result here.

After a few runs on 5090, I ended up with decent performance, but the setup only really made sense once I adjusted around router policy.

Curious if anyone else landed in the same place.

[kernelmike]

Not sure I agree there. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[batchsize]

Not sure I agree there. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Not saying you're wrong, just t

My first guess would be this: On B200, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[ggufwire22]

[quote]Not sure I agree there. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Not saying you're wrong,

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefix caching. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[localpatch78]

[quote][quote]Not sure I agree there. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Not saying you're

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tail latency. Once you know the plain path works, then add the faster path back in piece by piece.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[localpatch]

Ran a quick check on my box. On A6000 with Ollama, I saw basically the same behavior. The main thing that moved was latency, not quality.

[rocmpath]

Ran a quick check on my box. On A6000 with Ollama, I saw basically the same behavior. The main thing that moved was latency, not quality.

I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[localpatch78]

Interesting, I hadn't tried that. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[sparrowshed]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Phi and compare from there.