Posts: 50
Threads: 6
Joined: Sep 2026
Reputation:
0
This keeps coming up in conversation so I figured I'd start a thread.
The more I work with local setups, the more it seems like batch size decides everything downstream.
Wondering if this is just where the whole space is heading now.
For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.
Posts: 12
Threads: 2
Joined: Aug 2026
Reputation:
0
Maybe, but I had the opposite result. I've seen people blame prefix caching when the real issue was much more mundane. Worth testing the simpler explanation before adopting the more interesting one.
Posts: 142
Threads: 20
Joined: Sep 2026
Reputation:
0
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at INT4. Once you know the plain path works, then add the faster path back in piece by piece.
small models can still be useful
Posts: 57
Threads: 8
Joined: Oct 2026
Reputation:
0
Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.
If you post your exact settings, people here can probably narrow it down pretty fast.
Posts: 43
Threads: 5
Joined: Sep 2026
Reputation:
0
My first guess would be this: On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.
Posts: 28
Threads: 5
Joined: Sep 2026
Reputation:
0
Can you fill in a bit more context? Are you testing this on M3 Max or something else?
