+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Optimization & Performance (https://www.datasculpting.ai/inference/forumdisplay.php?fid=10)
+--- Forum: Performance Tuning (https://www.datasculpting.ai/inference/forumdisplay.php?fid=42)
+--- Thread: latency vs throughput for small deployments: anyone testing this? (/showthread.php?tid=795)
latency vs throughput for small deployments: anyone testing this? - papermixer4 - 10-06-2026
Did a bit of testing and thought I'd post the result here.
Short version: PyTorch behaved well for me, but only after I stopped ignoring tokens/sec.
Nothing scientific here, just practical notes from a real setup.
Re: latency vs throughput for small deployments: anyone testing this? - sparrowshed - 10-06-2026
same here on my side.