when video is still not worth the wait

[turbosupply83]

Appreciate the detail. The pattern around prefill/decode split has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[plainember52]

Good point. The pattern around batch size has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[visionforge]

Good point. The pattern around batch size has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at wh

subscribed, curious where this lands.

[cobalttinker]

Maybe, but I had the opposite result. I get why that seems right, but on Threadripper box the bottleneck felt elsewhere. Worth testing the simpler explanation before adopting the more interesting one.