10-06-2026, 01:39 AM
This keeps coming up in conversation so I figured I'd start a thread.
The more I work with local setups, the more it seems like prefix caching decides everything downstream.
Maybe obvious to some of you, but it clicked for me this week.
I care more about stability than squeezing the last possible tokens/sec out of it.
I expected the biggest difference to come from the model, but the stack around it mattered at least as much.
The more I work with local setups, the more it seems like prefix caching decides everything downstream.
Maybe obvious to some of you, but it clicked for me this week.
I care more about stability than squeezing the last possible tokens/sec out of it.
I expected the biggest difference to come from the model, but the stack around it mattered at least as much.