running local models on a phone

[fusedriver]

Appreciate the detail. I don't think your conclusion is far off.

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at quant drift.

[llmpeak]

Appreciate the detail. I don't think your conclusion is far off.

[torchkite]

Interesting, I hadn't tried that. I ended up in about the same place using PyTorch. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

If you post your exact settings, people here can probably narrow it down pretty fast.

[marbleyard32]

Interesting, I hadn't tried that. I ended up in about the same place using PyTorch. The biggest change for me was simplifying the stack instead of add

I don't think that's the whole story. I've seen people blame long context when the real issue was much more mundane. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[dockbrook21]

Appreciate the detail. I don't think your conclusion is far off.

That matches what I was seeing. The pattern around router policy has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[sparrowghost]

Appreciate the detail. I don't think your conclusion is far off.

subscribed, curious where this lands.

[fastquill74]

I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.

I'm happy to test one more variant if you want a second data point.

[phasescope]

Interesting, I hadn't tried that. I ended up in about the same place using vLLM.

[plainscope50]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Kimi and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.

If you post your exact settings, people here can probably narrow it down pretty fast.

[batchpeak]

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of PyTorch and 7900 XTX felt surprisingly usable once I tuned around continuous batching.

Maybe useful as a starting point for newer folks.

I care more about stability than squeezing the last possible tokens/sec out of it.