+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Optimization & Performance (https://www.datasculpting.ai/inference/forumdisplay.php?fid=10)
+--- Forum: Performance Tuning (https://www.datasculpting.ai/inference/forumdisplay.php?fid=42)
+--- Thread: latency vs throughput for small deployments (/showthread.php?tid=953)
latency vs throughput for small deployments - kernelrunner - 10-10-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
This isn't revolutionary, but the combination of LM Studio and GB200 felt surprisingly usable once I tuned around continuous batching.
Maybe useful as a starting point for newer folks.
Re: latency vs throughput for small deployments - papermixer4 - 10-10-2026
I tested this tonight. My result was mixed: first run looked bad, second run looked normal.
Re: latency vs throughput for small deployments - signalcraft14 - 10-10-2026
I'd start with the simpler stack.
Re: latency vs throughput for small deployments - papermixer4 - 10-10-2026
Maybe, but I had the opposite result. I've seen people blame token cost when the real issue was much more mundane.
Re: latency vs throughput for small deployments - bytestack - 10-12-2026
Quote:Maybe, but I had the opposite result. I've seen people blame token cost when the real issue was much more mundane.
What worked for me was pretty simple. On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Re: latency vs throughput for small deployments - rocmpath - 10-12-2026
I don't think that's the whole story. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the simpler explanation before adopting the more interesting one.
Re: latency vs throughput for small deployments - cudahouse - 10-13-2026
What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefill/decode split. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: latency vs throughput for small deployments - cudapeak - 10-13-2026
I hit almost the same issue last night.
Re: latency vs throughput for small deployments - stackarch - 10-13-2026
following this because I'm in the same boat. I keep promising myself this is the last rebuild.
Re: latency vs throughput for small deployments - macropilot - 10-14-2026
Good point. I don't think your conclusion is far off. The logs were somehow calmer than I was.