10-17-2026, 07:59 PM
Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Could be a workload difference, but I wouldn't generalize too fast from one setup.
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.
throughput matters, but tail latency matters too