09-27-2026, 02:28 AM
Posting this in case it saves someone else a little time.
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about tool calling.
Would rather learn from your scars than collect my own all at once.
I care more about stability than squeezing the last possible tokens/sec out of it.
I expected the biggest difference to come from the model, but the stack around it mattered at least as much.
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about tool calling.
Would rather learn from your scars than collect my own all at once.
I care more about stability than squeezing the last possible tokens/sec out of it.
I expected the biggest difference to come from the model, but the stack around it mattered at least as much.
throughput matters, but tail latency matters too