+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Optimization & Performance (https://www.datasculpting.ai/inference/forumdisplay.php?fid=10)
+--- Forum: Performance Tuning (https://www.datasculpting.ai/inference/forumdisplay.php?fid=42)
+--- Thread: latency vs throughput for small deployments on 7900 XTX (/showthread.php?tid=191)
Pages:
1
2
latency vs throughput for small deployments on 7900 XTX - bytestack - 09-08-2026
Posting this in case it saves someone else a little time.
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around SGLang and Threadripper box.
Would rather learn from your scars than collect my own all at once.
Re: latency vs throughput for small deployments on 7900 XTX - kernelmike - 09-08-2026
What exact setup are you running? What model, what runtime, and roughly what context size?
Re: latency vs throughput for small deployments on 7900 XTX - edgeriver - 09-08-2026
That matches what I was seeing. That matches what I saw on 5090. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: latency vs throughput for small deployments on 7900 XTX - batchscope72 - 09-08-2026
Quote:That matches what I was seeing. That matches what I saw on 5090. The biggest change for me was simplifying the stack instead of adding one more optimi
What exact setup are you running? Are you testing this on B200 or something else?
Re: latency vs throughput for small deployments on 7900 XTX - rasterpoint - 09-08-2026
Thanks, that helps. I ended up in about the same place using llama.cpp. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
If you post your exact settings, people here can probably narrow it down pretty fast.
Re: latency vs throughput for small deployments on 7900 XTX - tensorfield88 - 09-10-2026
same here on my side.
Re: latency vs throughput for small deployments on 7900 XTX - sparrowshed - 09-10-2026
I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly.
Re: latency vs throughput for small deployments on 7900 XTX - tensorfield88 - 09-12-2026
Quote:I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly.
What exact setup are you running? Are you testing this on 5090 or something else?
Re: latency vs throughput for small deployments on 7900 XTX - kernelmike - 09-13-2026
Quote:Quote:I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly.
Interesting, I hadn't tried that. The pattern around continuous batching has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: latency vs throughput for small deployments on 7900 XTX - localpatch - 09-13-2026
What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at quant drift. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it. Turns out 'just one more test' was a lie.