continuous batching wins and losses

[promptpoint]

Appreciate the detail. The pattern around tail latency has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[batchsize]

Thanks, that helps. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[ggufvault]

Can you fill in a bit more context? Is this local-only or are you routing anything externally?

[nanogarden]

Need a little more detail here. Are you testing this on 7900 XTX or something else?

[mangofield33]

Need a little more detail here. Are you testing this on 7900 XTX or something else?

Maybe, but I had the opposite result. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Could be a workload difference, but I wouldn't generalize too fast from one setup.