+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Apps / Utilities / Tools (https://www.datasculpting.ai/inference/forumdisplay.php?fid=18)
+--- Forum: Local (https://www.datasculpting.ai/inference/forumdisplay.php?fid=20)
+--- Thread: desktop apps around local AI for Phi (/showthread.php?tid=847)
Pages:
1
2
desktop apps around local AI for Phi - papermixer4 - 10-07-2026
Spent some time on this today and wanted to compare notes.
I've gone back and forth between PyTorch and SGLang for Mistral. The interesting difference for me has been INT4.
Interested in tradeoffs more than fan loyalty.
Re: desktop apps around local AI for Phi - rocmriley - 10-07-2026
Thanks, that helps. The pattern around batch size has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: desktop apps around local AI for Phi - audionode8 - 10-08-2026
that fixed it for me too.
Re: desktop apps around local AI for Phi - mlxgarden - 10-08-2026
Can you fill in a bit more context? What model, what runtime, and roughly what context size?
Re: desktop apps around local AI for Phi - mintstation99 - 10-09-2026
My first guess would be this: If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.
Re: desktop apps around local AI for Phi - ggufwire22 - 10-09-2026
I think the bottleneck is somewhere else. I've seen people blame prefix caching when the real issue was much more mundane. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.
Re: desktop apps around local AI for Phi - vectorport - 10-10-2026
Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.
Re: desktop apps around local AI for Phi - mlxgarden - 10-10-2026
I don't think that's the whole story. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.
Re: desktop apps around local AI for Phi - dockember - 10-10-2026
Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?
Re: desktop apps around local AI for Phi - vramstream - 10-11-2026
worth checking the driver first.