video model setup pain points

[northhall88]

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about consumer GPUs.

I'm especially interested in mistakes worth avoiding.

[kernelmike]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at speculative decoding. Once you know the plain path works, then add the faster path back in piece by piece.

[mangofield33]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at speculative decoding. Once

Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?

[thetanest]

Maybe, but I had the opposite result. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.

[llm_wren]

That matches what I was seeing. I ended up in about the same place using vLLM. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[syshouse]

That matches what I was seeing. I ended up in about the same place using vLLM. The biggest change for me was simplifying the stack instead of adding o

A couple questions before I guess too hard. Is this local-only or are you routing anything externally?

[phasetrack]

following this because I'm in the same boat.

[vectorport]

Appreciate the detail. That matches what I saw on GB200. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[rocmriley]

Appreciate the detail. The pattern around HBM constraints has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[quiettoken]

What exact setup are you running? Is this local-only or are you routing anything externally?