09-27-2026, 06:29 PM
This keeps coming up in conversation so I figured I'd start a thread.
The more I work with local setups, the more it seems like tail latency decides everything downstream.
Curious whether that lines up with what other people are seeing.
I care more about stability than squeezing the last possible tokens/sec out of it.
The more I work with local setups, the more it seems like tail latency decides everything downstream.
Curious whether that lines up with what other people are seeing.
I care more about stability than squeezing the last possible tokens/sec out of it.