desktop apps around local AI

[quiettoken]

Spent some time on this today and wanted to compare notes.

Two paths both seem viable, but they break down differently once speculative decoding becomes important.

Interested in tradeoffs more than fan loyalty.

[thetaplane]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.

[vectorcrate98]

That matches what I was seeing. That matches what I saw on dual 3090.