latency vs throughput for small deployments with PyTorch

[homelabhenry]

[quote][quote][quote]Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. The main t

Interesting, I hadn't tried that. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[sparrowshed]

I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher.

[signalcraft14]

Maybe, but I had the opposite result. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Worth testing the simpler explanation before adopting the more interesting one.