Used A6000 worth it now: anyone testing this?

[syshouse]

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like tail latency decides everything downstream.

Curious whether that lines up with what other people are seeing.

[vrambandit]

I don't think that's the whole story. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[sanevault]

I hit almost the same issue last night.

[paperrunner]

that fixed it for me too.

[rocmriley]

that fixed it for me too.

[cudapoint]

Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?

[paperrunner]

Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?

A couple questions before I guess too hard. Are you testing this on 7900 XTX or something else?