latency vs throughput for small deployments on Threadripper box

[tensorport]

Did a bit of testing and thought I'd post the result here.

After a few runs on B200, I ended up with decent performance, but the setup only really made sense once I adjusted around tokens/sec.

Nothing scientific here, just practical notes from a real setup. At this point the real bottleneck is my patience.

[tensorfield88]

Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher.

[signalcraft14]

Thanks, that helps. That matches what I saw on GB200. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[sparrowshed]

Thanks, that helps. That matches what I saw on GB200. I got better results once I stopped chasing one metric and just looked at whether the workflow f

Appreciate the detail. That matches what I saw on mini PC.

[turboscope5]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at router policy. Not glamorous advice, but repeatable baselines save a lot of time.

[kernelrunner]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at router policy. Not glamo

Good point. That matches what I saw on 5090. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[localpatch]

A couple questions before I guess too hard. Are you testing this on A6000 or something else?