latency vs throughput for small deployments on Threadripper box

[tensorport]

Did a bit of testing and thought I'd post the result here.

After a few runs on B200, I ended up with decent performance, but the setup only really made sense once I adjusted around tokens/sec.

Nothing scientific here, just practical notes from a real setup. At this point the real bottleneck is my patience.