+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Optimization & Performance (https://www.datasculpting.ai/inference/forumdisplay.php?fid=10)
+--- Forum: Performance Tuning (https://www.datasculpting.ai/inference/forumdisplay.php?fid=42)
+--- Thread: latency vs throughput for small deployments on Threadripper box (/showthread.php?tid=93)
latency vs throughput for small deployments on Threadripper box - tensorport - 09-01-2026
Did a bit of testing and thought I'd post the result here.
After a few runs on B200, I ended up with decent performance, but the setup only really made sense once I adjusted around tokens/sec.
Nothing scientific here, just practical notes from a real setup. At this point the real bottleneck is my patience.
Re: latency vs throughput for small deployments on Threadripper box - tensorfield88 - 09-01-2026
Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher.
Re: latency vs throughput for small deployments on Threadripper box - signalcraft14 - 09-02-2026
Thanks, that helps. That matches what I saw on GB200. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: latency vs throughput for small deployments on Threadripper box - sparrowshed - 09-02-2026
Quote:Thanks, that helps. That matches what I saw on GB200. I got better results once I stopped chasing one metric and just looked at whether the workflow f
Appreciate the detail. That matches what I saw on mini PC.
Re: latency vs throughput for small deployments on Threadripper box - turboscope5 - 09-02-2026
I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at router policy. Not glamorous advice, but repeatable baselines save a lot of time.
Re: latency vs throughput for small deployments on Threadripper box - kernelrunner - 09-03-2026
Quote:I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at router policy. Not glamo
Good point. That matches what I saw on 5090. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: latency vs throughput for small deployments on Threadripper box - localpatch - 09-03-2026
A couple questions before I guess too hard. Are you testing this on A6000 or something else?