Posts: 142
Threads: 20
Joined: Sep 2026
Reputation:
0
This keeps coming up in conversation so I figured I'd start a thread.
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with token cost.
Curious whether that lines up with what other people are seeing.
This is mostly for local work rather than public serving.
small models can still be useful
Posts: 72
Threads: 7
Joined: Sep 2026
Reputation:
0
A couple questions before I guess too hard. Is this local-only or are you routing anything externally?
Posts: 142
Threads: 20
Joined: Sep 2026
Reputation:
0
Not sure I agree there. I get why that seems right, but on M3 Max the bottleneck felt elsewhere. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.
small models can still be useful
Posts: 53
Threads: 4
Joined: Sep 2026
Reputation:
0
I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Mixtral and compare from there.
Posts: 50
Threads: 6
Joined: Sep 2026
Reputation:
0
following this because I'm in the same boat.
Posts: 42
Threads: 1
Joined: Oct 2026
Reputation:
0
What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Llama and compare from there.
Posts: 57
Threads: 8
Joined: Oct 2026
Reputation:
0
A couple things I'd check first. If you're on Ollama, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.
Posts: 72
Threads: 7
Joined: Sep 2026
Reputation:
0
subscribed, curious where this lands.
Posts: 93
Threads: 10
Joined: Oct 2026
Reputation:
0
Good point. That matches what I saw on A6000. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
