desktop apps around local AI

[rocmriley]

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like prefix caching decides everything downstream.

Maybe obvious to some of you, but it clicked for me this week.

[mlxgarden]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at router policy.

[relayforge]

Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.