Any point in NPUs yet for Phi

[quantyard]

Posting this in case it saves someone else a little time.

I'm not trying to build a perfect architecture diagram. I just want a sane starting point around MLX and M3 Max.

I'm especially interested in mistakes worth avoiding.

This is mostly for local work rather than public serving.

[northgarden37]

Posting this in case it saves someone else a little time.

Maybe, but I had the opposite result. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the simpler explanation before adopting the more interesting one. At this point the real bottleneck is my patience.

[dockember]

Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher.

[torchkite]

Appreciate the detail. I ended up in about the same place using MLX. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[visionforge]

What exact setup are you running? Are you testing this on GB200 or something else?

[sparrowcraft]

What exact setup are you running? Are you testing this on GB200 or something else?

following this because I'm in the same boat.

[llmarch83]

worth checking the driver first.