+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Hardware (https://www.datasculpting.ai/inference/forumdisplay.php?fid=4)
+--- Forum: CPUs / NPUs / Accelerators (https://www.datasculpting.ai/inference/forumdisplay.php?fid=30)
+--- Thread: Any point in NPUs yet for Phi (/showthread.php?tid=20)
Any point in NPUs yet for Phi - quantyard - 08-23-2026
Posting this in case it saves someone else a little time.
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around MLX and M3 Max.
I'm especially interested in mistakes worth avoiding.
This is mostly for local work rather than public serving.
Re: Any point in NPUs yet for Phi - northgarden37 - 08-25-2026
Quote:Posting this in case it saves someone else a little time.
Maybe, but I had the opposite result. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the simpler explanation before adopting the more interesting one. At this point the real bottleneck is my patience.
Re: Any point in NPUs yet for Phi - dockember - 08-26-2026
Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher.
Re: Any point in NPUs yet for Phi - torchkite - 08-28-2026
Appreciate the detail. I ended up in about the same place using MLX. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: Any point in NPUs yet for Phi - visionforge - 08-28-2026
What exact setup are you running? Are you testing this on GB200 or something else?
Re: Any point in NPUs yet for Phi - sparrowcraft - 08-28-2026
Quote:What exact setup are you running? Are you testing this on GB200 or something else?
following this because I'm in the same boat.
Re: Any point in NPUs yet for Phi - llmarch83 - 08-29-2026
worth checking the driver first.