+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Optimization & Performance (https://www.datasculpting.ai/inference/forumdisplay.php?fid=10)
+--- Forum: Performance Tuning (https://www.datasculpting.ai/inference/forumdisplay.php?fid=42)
+--- Thread: latency vs throughput for small deployments with PyTorch (/showthread.php?tid=396)
Pages:
1
2
latency vs throughput for small deployments with PyTorch - amberrunner - 09-21-2026
Curious what people here are seeing.
My current setup is TensorRT-LLM with M3 Max, and I keep running into KV cache pressure.
If you've tested something similar, what did you keep and what did you stop bothering with?
For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.
Re: latency vs throughput for small deployments with PyTorch - vramport - 09-22-2026
I hit almost the same issue last night.
Re: latency vs throughput for small deployments with PyTorch - rocmpath - 09-23-2026
same here on my side.
Re: latency vs throughput for small deployments with PyTorch - tensorfield88 - 09-23-2026
Quote:same here on my side.
worth checking the driver first.
Re: latency vs throughput for small deployments with PyTorch - papermixer4 - 09-23-2026
Quote:Quote:same here on my side.
I'd start with the simpler stack.
Re: latency vs throughput for small deployments with PyTorch - localpatch - 09-24-2026
same here on my side.
Re: latency vs throughput for small deployments with PyTorch - kernelmike - 09-24-2026
Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.
Re: latency vs throughput for small deployments with PyTorch - tinycontext - 09-25-2026
Quote:Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was l
My first guess would be this: Before rebuilding anything, I'd test one clean baseline with FLUX and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.
Re: latency vs throughput for small deployments with PyTorch - kernelmike - 09-26-2026
Quote:[quote]Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that move
Good point. I ended up in about the same place using vLLM. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: latency vs throughput for small deployments with PyTorch - signalcraft14 - 09-28-2026
Quote:[quote][quote]Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. The main thing th
My first guess would be this: If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it. At this point the real bottleneck is my patience.