industry move that matters for local users: early impressions

[edgecoast]

Posting this in case it saves someone else a little time.

I'm not trying to build a perfect architecture diagram. I just want a sane starting point around llama.cpp and M3 Max.

I'm especially interested in mistakes worth avoiding.

I care more about stability than squeezing the last possible tokens/sec out of it.

The part that surprised me was how much the boring details mattered more than the model choice.

[benchplease]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at embeddings.

[amberrunner]

Good point. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

I'm happy to test one more variant if you want a second data point.

[mangowire]

Thanks, that helps. I don't think your conclusion is far off.

[modbench]

Can you fill in a bit more context? Are you testing this on mini PC or something else?

[vramscope]

Thanks, that helps. I ended up in about the same place using PyTorch.

[routearch]

Thanks, that helps. I ended up in about the same place using PyTorch.

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at token cost. Not glamorous advice, but repeatable baselines save a lot of time.

[deepseeked]

that fixed it for me too.

[audiopoint18]

Can you fill in a bit more context? Is this local-only or are you routing anything externally?

[localtinker]

What worked for me was pretty simple. If you're on Ollama, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.