+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Apps / Utilities / Tools (https://www.datasculpting.ai/inference/forumdisplay.php?fid=18)
+--- Forum: Local (https://www.datasculpting.ai/inference/forumdisplay.php?fid=20)
+--- Thread: little utility you keep installed (/showthread.php?tid=567)
Pages:
1
2
little utility you keep installed - mlxgarden - 09-29-2026
This keeps coming up in conversation so I figured I'd start a thread.
One thing I keep noticing is that people underestimate continuous batching until they actually try to run this stuff for more than one afternoon.
Curious whether that lines up with what other people are seeing.
Re: little utility you keep installed - mangoquill - 09-29-2026
Did a small comparison this afternoon. On 5090 with Ollama, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.
Re: little utility you keep installed - vramstream - 09-29-2026
Ran a quick check on my box. On A6000 with TensorRT-LLM, I saw basically the same behavior.
Re: little utility you keep installed - relayforge - 10-01-2026
That matches what I was seeing. That matches what I saw on Threadripper box.
Re: little utility you keep installed - quiettoken - 10-01-2026
same here on my side.
Re: little utility you keep installed - vectorport - 10-01-2026
Quote:same here on my side.
that fixed it for me too.
Re: little utility you keep installed - audionode8 - 10-04-2026
That matches what I was seeing. The pattern around router policy has been pretty similar for me.
Re: little utility you keep installed - rocmcrate26 - 10-07-2026
Quote:That matches what I was seeing. The pattern around router policy has been pretty similar for me.
Ran a quick check on my box. On 4090 with SGLang, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.
Re: little utility you keep installed - rocmhouse90 - 10-07-2026
that fixed it for me too.
Re: little utility you keep installed - mlxgarden - 10-09-2026
same here on my side.