Audio / Speech discussion

[ggufvault]

Posting this in case it saves someone else a little time.

If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

I'm especially interested in mistakes worth avoiding.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

[flintriver50]

That matches what I was seeing. That matches what I saw on 7900 XTX.

[tensorfield88]

That matches what I was seeing. That matches what I saw on 7900 XTX.

Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal.

[vramforge]

That matches what I was seeing. That matches what I saw on 7900 XTX.

Interesting, I hadn't tried that. I ended up in about the same place using SGLang. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

I'm happy to test one more variant if you want a second data point.

[signalpeak]

That matches what I was seeing. That matches what I saw on 7900 XTX.

that fixed it for me too.

[cudapoint]

That matches what I was seeing. That matches what I saw on 7900 XTX.

Can you fill in a bit more context? Are you testing this on B200 or something else?

[relayforge]

I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher.

[kernelhouse70]

A couple questions before I guess too hard. What model, what runtime, and roughly what context size?

[signalcraft14]

A couple questions before I guess too hard. Are you testing this on A6000 or something else?

[flintriver50]

I don't think that's the whole story. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.