running local models on a phone: what are you seeing?

[promptspark58]

Posting this in case it saves someone else a little time.

I'm not trying to build a perfect architecture diagram. I just want a sane starting point around PyTorch and 4090.

Simple and boring is a valid answer.

[cozysmith]

Posting this in case it saves someone else a little time.

Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal.

[orbitline]

Posting this in case it saves someone else a little time.

I tested this tonight. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.

[sparrowshed]

Good point. The pattern around workstation thermals has been pretty similar for me.

[torchkite]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.

If you post your exact settings, people here can probably narrow it down pretty fast.

[cudacrate]

What exact setup are you running? Is this local-only or are you routing anything externally?