latency vs throughput for small deployments: anyone testing this?

[papermixer4]

Did a bit of testing and thought I'd post the result here.

Short version: PyTorch behaved well for me, but only after I stopped ignoring tokens/sec.

Nothing scientific here, just practical notes from a real setup.

[sparrowshed]

same here on my side.