Posts: 49
Threads: 7
Joined: Oct 2026
Reputation:
0
Maybe I'm missing something obvious.
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling reranking.
Not looking for the perfect answer, mostly what worked in real use.
trying not to rebuild again
Posts: 137
Threads: 16
Joined: Sep 2026
Reputation:
0
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at KV cache pressure. Not glamorous advice, but repeatable baselines save a lot of time.
I'm happy to test one more variant if you want a second data point.
throughput matters, but tail latency matters too
Posts: 24
Threads: 1
Joined: Sep 2026
Reputation:
0
A couple things I'd check first. On A6000, the easiest win is often to back off the aggressive settings and see whether the problem survives.
learning in public
Posts: 199
Threads: 27
Joined: Sep 2026
Reputation:
0
following this because I'm in the same boat. I keep promising myself this is the last rebuild.
M3 Max • MLX • local-first
Posts: 44
Threads: 3
Joined: Oct 2026
Reputation:
0
Maybe, but I had the opposite result. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Worth testing the simpler explanation before adopting the more interesting one.
Posts: 137
Threads: 16
Joined: Sep 2026
Reputation:
0
What exact setup are you running? Is this local-only or are you routing anything externally?
throughput matters, but tail latency matters too
Posts: 49
Threads: 7
Joined: Oct 2026
Reputation:
0
Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher.
trying not to rebuild again
Posts: 17
Threads: 1
Joined: Sep 2026
Reputation:
0
worth checking the driver first.
Posts: 44
Threads: 3
Joined: Oct 2026
Reputation:
0
I hit almost the same issue last night.
Posts: 73
Threads: 5
Joined: Aug 2026
Reputation:
0
subscribed, curious where this lands.
