running local models on a phone with PyTorch

[plainline]

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about quant drift.

Would rather learn from your scars than collect my own all at once.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

[sparrowshed]

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with FLUX and compare from there.

[relaysupply90]

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with FLUX and compare from there.

Good point. That matches what I saw on GB200. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[turbosupply83]

worth checking the driver first.

[cachebench]

Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. I expected the model choice to matter more than it did.

[cudacrate]

Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. I expected the model choice to

I hit almost the same issue last night.

[sparrowshed]

A couple things I'd check first. If you're on Ollama, I'd confirm the version first and then reduce variables until the behavior is obvious.

[fusedriver]

Appreciate the detail. That matches what I saw on B200.

[sparrowworks88]

that fixed it for me too.

[plainline]

A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?