when video is still not worth the wait

[audiopath66]

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like token cost decides everything downstream.

Curious whether that lines up with what other people are seeing.

This is mostly for local work rather than public serving.

[quantghost]

Thanks, that helps. That matches what I saw on A6000. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[batchwire]

that fixed it for me too.

[plainworks]

Thanks, that helps. I don't think your conclusion is far off.

[sparrowghost]

That matches what I was seeing. The pattern around long context has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[latentcraft]

following this because I'm in the same boat.

If you post your exact settings, people here can probably narrow it down pretty fast.

[quiethouse]

Maybe, but I had the opposite result. I get why that seems right, but on B200 the bottleneck felt elsewhere. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[plainline]

A couple questions before I guess too hard. Are you testing this on M3 Max or something else?

[quiettoken]

A couple questions before I guess too hard. Are you testing this on M3 Max or something else?

Appreciate the detail. I ended up in about the same place using PyTorch. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[mangovault67]

Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.