+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Optimization & Performance (https://www.datasculpting.ai/inference/forumdisplay.php?fid=10)
+--- Forum: Performance Tuning (https://www.datasculpting.ai/inference/forumdisplay.php?fid=42)
+--- Thread: latency vs throughput for small deployments with PyTorch (/showthread.php?tid=396)
Pages:
1
2
Re: latency vs throughput for small deployments with PyTorch - homelabhenry - 09-29-2026
Quote:[quote][quote][quote]Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. The main t
Interesting, I hadn't tried that. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: latency vs throughput for small deployments with PyTorch - sparrowshed - 10-02-2026
I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher.
Re: latency vs throughput for small deployments with PyTorch - signalcraft14 - 10-04-2026
Maybe, but I had the opposite result. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Worth testing the simpler explanation before adopting the more interesting one.