+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Miscellaneous Discussion (https://www.datasculpting.ai/inference/forumdisplay.php?fid=24)
+--- Forum: News & Industry (https://www.datasculpting.ai/inference/forumdisplay.php?fid=26)
+--- Thread: industry move that matters for local users: early impressions (/showthread.php?tid=611)
industry move that matters for local users: early impressions - edgecoast - 09-30-2026
Posting this in case it saves someone else a little time.
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around llama.cpp and M3 Max.
I'm especially interested in mistakes worth avoiding.
I care more about stability than squeezing the last possible tokens/sec out of it.
The part that surprised me was how much the boring details mattered more than the model choice.
Re: industry move that matters for local users: early impressions - benchplease - 10-01-2026
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at embeddings.
Re: industry move that matters for local users: early impressions - amberrunner - 10-01-2026
Good point. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
I'm happy to test one more variant if you want a second data point.
Re: industry move that matters for local users: early impressions - mangowire - 10-01-2026
Thanks, that helps. I don't think your conclusion is far off.
Re: industry move that matters for local users: early impressions - modbench - 10-02-2026
Can you fill in a bit more context? Are you testing this on mini PC or something else?
Re: industry move that matters for local users: early impressions - vramscope - 10-02-2026
Thanks, that helps. I ended up in about the same place using PyTorch.
Re: industry move that matters for local users: early impressions - routearch - 10-03-2026
Quote:Thanks, that helps. I ended up in about the same place using PyTorch.
I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at token cost. Not glamorous advice, but repeatable baselines save a lot of time.
Re: industry move that matters for local users: early impressions - deepseeked - 10-03-2026
that fixed it for me too.
Re: industry move that matters for local users: early impressions - audiopoint18 - 10-03-2026
Can you fill in a bit more context? Is this local-only or are you routing anything externally?
Re: industry move that matters for local users: early impressions - localtinker - 10-04-2026
What worked for me was pretty simple. If you're on Ollama, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.