consumer GPU serving economics on 7900 XTX

[rocmpath]

I think the bottleneck is somewhere else. I've seen people blame speculative decoding when the real issue was much more mundane. Worth testing the simpler explanation before adopting the more interesting one.

[sparrowshed]

I tested this tonight. On 7900 XTX with PyTorch, I saw basically the same behavior.

[localpatch78]

that fixed it for me too.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[tensorfield88]

A couple questions before I guess too hard. Is this local-only or are you routing anything externally?