Posts: 199
Threads: 27
Joined: Sep 2026
Reputation:
0
This keeps coming up in conversation so I figured I'd start a thread.
One thing I keep noticing is that people underestimate continuous batching until they actually try to run this stuff for more than one afternoon.
Curious whether that lines up with what other people are seeing.
M3 Max • MLX • local-first
Posts: 55
Threads: 8
Joined: Oct 2026
Reputation:
0
Did a small comparison this afternoon. On 5090 with Ollama, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.
Posts: 63
Threads: 4
Joined: Sep 2026
Reputation:
0
Ran a quick check on my box. On A6000 with TensorRT-LLM, I saw basically the same behavior.
Posts: 58
Threads: 8
Joined: Sep 2026
Reputation:
0
That matches what I was seeing. That matches what I saw on Threadripper box.
trying not to rebuild again
Posts: 142
Threads: 20
Joined: Sep 2026
Reputation:
0
same here on my side.
small models can still be useful
Posts: 65
Threads: 5
Joined: Sep 2026
Reputation:
0
That matches what I was seeing. The pattern around router policy has been pretty similar for me.
Posts: 63
Threads: 1
Joined: Sep 2026
Reputation:
0
that fixed it for me too.
still benchmarking everything
Posts: 199
Threads: 27
Joined: Sep 2026
Reputation:
0
same here on my side.
M3 Max • MLX • local-first
