+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: vLLM / SGLang (https://www.datasculpting.ai/inference/forumdisplay.php?fid=36)
+--- Thread: why did throughput drop after update (/showthread.php?tid=479)
why did throughput drop after update - cacheline - 09-25-2026
Posting this in case it saves someone else a little time.
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around MLX and mini PC.
I'm especially interested in mistakes worth avoiding.
My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.
Re: why did throughput drop after update - routecraft - 09-25-2026
I'd approach it in a boring order. If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.
Re: why did throughput drop after update - cacheline - 09-25-2026
That matches what I was seeing. The pattern around quant drift has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: why did throughput drop after update - mangofield33 - 09-25-2026
following this because I'm in the same boat.
Re: why did throughput drop after update - cacheline - 09-25-2026
Quote:following this because I'm in the same boat.
A couple questions before I guess too hard. What model, what runtime, and roughly what context size?
Re: why did throughput drop after update - thetaplane - 09-25-2026
I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. I expected the model choice to matter more than it did.
Re: why did throughput drop after update - turboarch95 - 09-27-2026
Appreciate the detail. I ended up in about the same place using PyTorch. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: why did throughput drop after update - thetagrid15 - 09-27-2026
A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Llama and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.
Re: why did throughput drop after update - batchsize - 09-27-2026
same here on my side.
Re: why did throughput drop after update - promptpoint - 09-27-2026
A couple questions before I guess too hard. What model, what runtime, and roughly what context size?