Posts: 36
Threads: 7
Joined: Sep 2026
Reputation:
0
This keeps coming up in conversation so I figured I'd start a thread.
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with tool calling.
Wondering if this is just where the whole space is heading now.
Posts: 31
Threads: 2
Joined: Oct 2026
Reputation:
0
I'd approach it in a boring order. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.
Posts: 28
Threads: 3
Joined: Aug 2026
Reputation:
0
I tested this tonight. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.
Posts: 44
Threads: 3
Joined: Oct 2026
Reputation:
0
Ran a quick check on my box. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.
Posts: 112
Threads: 12
Joined: Sep 2026
Reputation:
0
Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.
I'm happy to test one more variant if you want a second data point.
context windows are not a product strategy
Posts: 19
Threads: 5
Joined: Aug 2026
Reputation:
0
that fixed it for me too.
If you post your exact settings, people here can probably narrow it down pretty fast.
Posts: 40
Threads: 1
Joined: Sep 2026
Reputation:
0
My first guess would be this: On M3 Max, the easiest win is often to back off the aggressive settings and see whether the problem survives.
learning in public
Posts: 48
Threads: 5
Joined: Sep 2026
Reputation:
0
I'd approach it in a boring order. On A6000, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.
