latency vs throughput for small deployments: setup advice

[stackarch]

Appreciate the detail. I ended up in about the same place using PyTorch. Worth rechecking your defaults though, because one bad assumption can make th

Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.