70B still worth the pain for Phi

[quietsupply11]

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on mini PC, or squeeze more out of the stack and deal with the complexity.

Interested in tradeoffs more than fan loyalty.

This is mostly for local work rather than public serving. The logs were somehow calmer than I was.

[audiorunner74]

Good point. That matches what I saw on A6000. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[visionforge]

Good point. That matches what I saw on A6000. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usab

That matches what I was seeing. That matches what I saw on 5090. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[vramstream]

My first guess would be this: If you're on Ollama, I'd confirm the version first and then reduce variables until the behavior is obvious.

[mellowsupply]

worth checking the driver first.

[batchshed]

That matches what I was seeing. I don't think your conclusion is far off.

[localtinker]

A couple things I'd check first. If you're on MLX, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.

[deltastation]

That matches what I was seeing. I don't think your conclusion is far off.

[plaingarden18]

Can you fill in a bit more context? Is this local-only or are you routing anything externally?

[kernelmike]

Thanks, that helps. That matches what I saw on Threadripper box. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.