09-15-2026, 06:04 AM
Did a bit of testing and thought I'd post the result here.
After a few runs on mini PC, I ended up with decent performance, but the setup only really made sense once I adjusted around INT4.
Curious if anyone else landed in the same place.
I expected the biggest difference to come from the model, but the stack around it mattered at least as much.
After a few runs on mini PC, I ended up with decent performance, but the setup only really made sense once I adjusted around INT4.
Curious if anyone else landed in the same place.
I expected the biggest difference to come from the model, but the stack around it mattered at least as much.
throughput matters, but tail latency matters too