+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Optimization & Performance (https://www.datasculpting.ai/inference/forumdisplay.php?fid=10)
+--- Forum: Performance Tuning (https://www.datasculpting.ai/inference/forumdisplay.php?fid=42)
+--- Thread: latency vs throughput for small deployments: setup advice (/showthread.php?tid=220)
Pages:
1
2
latency vs throughput for small deployments: setup advice - macropilot - 09-11-2026
This keeps coming up in conversation so I figured I'd start a thread.
The more I work with local setups, the more it seems like token cost decides everything downstream.
Wondering if this is just where the whole space is heading now.
Re: latency vs throughput for small deployments: setup advice - cacheworks - 09-11-2026
What exact setup are you running? What model, what runtime, and roughly what context size?
Re: latency vs throughput for small deployments: setup advice - prompttinker44 - 09-11-2026
Need a little more detail here. Is this local-only or are you routing anything externally?
Re: latency vs throughput for small deployments: setup advice - kernelrunner - 09-11-2026
Need a little more detail here. Are you testing this on M3 Max or something else?
Re: latency vs throughput for small deployments: setup advice - localpatch78 - 09-11-2026
Can you fill in a bit more context? Are you testing this on 5090 or something else?
Re: latency vs throughput for small deployments: setup advice - plainline - 09-11-2026
A couple things I'd check first. If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.
Re: latency vs throughput for small deployments: setup advice - signalcraft14 - 09-12-2026
I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at speculative decoding. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: latency vs throughput for small deployments: setup advice - vramport - 09-13-2026
Appreciate the detail. I ended up in about the same place using PyTorch. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: latency vs throughput for small deployments: setup advice - stackarch - 09-13-2026
Quote:Appreciate the detail. I ended up in about the same place using PyTorch. Worth rechecking your defaults though, because one bad assumption can make th
Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.
Re: latency vs throughput for small deployments: setup advice - edgeriver - 09-14-2026
What exact setup are you running? Are you testing this on mini PC or something else?