Help with Audio / Speech

[thetarunner]

Maybe I'm missing something obvious.

I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling batch size.

If you've tested something similar, what did you keep and what did you stop bothering with?

[quantyard]

Maybe, but I had the opposite result. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet. Turns out 'just one more test' was a lie.

[batchpeak]

I don't think that's the whole story. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[vectorfield]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at router policy. Once you know the plain path works, then add the faster path back in piece by piece.

[quietarch]

I'd approach it in a boring order. On A6000, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[bytesupply76]

I'd approach it in a boring order. On A6000, the easiest win is often to back off the aggressive settings and see whether the problem survives.

A couple things I'd check first. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

[signalpeak]

Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?

[flintriver50]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefill/decode split.

[sparrowcraft]

Interesting, I hadn't tried that. I don't think your conclusion is far off.