latency vs throughput for small deployments on 7900 XTX

[bytestack]

Posting this in case it saves someone else a little time.

I'm not trying to build a perfect architecture diagram. I just want a sane starting point around SGLang and Threadripper box.

Would rather learn from your scars than collect my own all at once.

[kernelmike]

What exact setup are you running? What model, what runtime, and roughly what context size?

[edgeriver]

That matches what I was seeing. That matches what I saw on 5090. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[batchscope72]

That matches what I was seeing. That matches what I saw on 5090. The biggest change for me was simplifying the stack instead of adding one more optimi

What exact setup are you running? Are you testing this on B200 or something else?

[rasterpoint]

Thanks, that helps. I ended up in about the same place using llama.cpp. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

If you post your exact settings, people here can probably narrow it down pretty fast.

[tensorfield88]

same here on my side.

[sparrowshed]

I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly.

[tensorfield88]

I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly.

What exact setup are you running? Are you testing this on 5090 or something else?

[kernelmike]

I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly.

Interesting, I hadn't tried that. The pattern around continuous batching has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[localpatch]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at quant drift. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it. Turns out 'just one more test' was a lie.