09-04-2026, 10:30 PM
This keeps coming up in conversation so I figured I'd start a thread.
The more I work with local setups, the more it seems like tokens/sec decides everything downstream.
Curious whether that lines up with what other people are seeing.
This is mostly for local work rather than public serving.
The more I work with local setups, the more it seems like tokens/sec decides everything downstream.
Curious whether that lines up with what other people are seeing.
This is mostly for local work rather than public serving.
throughput matters, but tail latency matters too