what made local AI click for you

[rocmrunner40]

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on M3 Max, or squeeze more out of the stack and deal with the complexity.

Interested in tradeoffs more than fan loyalty.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

The part that surprised me was how much the boring details mattered more than the model choice.

[ggufwire22]

I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet. I keep promising myself this is the last rebuild.

[stackarch]

I hit almost the same issue last night.

[quiettoken]

I think the bottleneck is somewhere else. I've seen people blame prefill/decode split when the real issue was much more mundane. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[plaingarden18]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Gemma and compare from there.

[marbletrack]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Gemma and compare from there.

I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.

[quiettoken]

Good point. The pattern around structured output has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[vectorshed43]

I don't think that's the whole story. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[tensorkite]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Qwen and compare from there.

[quiettoken]

Maybe, but I had the opposite result. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Worth testing the simpler explanation before adopting the more interesting one.