Inference › Optimization & Performance › Performance Tuning v
latency vs throughput for small deployments: setup advice

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Threaded Mode
latency vs throughput for small deployments: setup advice
localpatch78 Offline
Junior Member
**
Posts: 37
Threads: 3
Joined: Sep 2026
Reputation: 0
#11
09-14-2026, 04:59 AM
subscribed, curious where this lands.
Find
Reply
localpatch Offline
Junior Member
**
Posts: 36
Threads: 3
Joined: Sep 2026
Reputation: 0
#12
09-14-2026, 05:44 PM
Appreciate the detail. I ended up in about the same place using SGLang. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
still benchmarking everything
Find
Reply
kernelmike Offline
Member
***
Posts: 184
Threads: 23
Joined: Sep 2026
Reputation: 0
#13
09-16-2026, 06:01 AM
same here on my side.
Mostly CUDA, sometimes wrong, usually measured
Find
Reply


Forum Jump:


Users browsing this thread: 1 Guest(s)