when video is still not worth the wait: anyone testing this?

[marbletrack]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Llama and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[plaingarden18]

I think the bottleneck is somewhere else. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.

[oxideworks59]

Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.

[cacheline]

What exact setup are you running? Do you see the same thing with a smaller batch or shorter prompt?