+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: vLLM / SGLang (https://www.datasculpting.ai/inference/forumdisplay.php?fid=36)
+--- Thread: why did throughput drop after update: early impressions (/showthread.php?tid=1249)
Pages:
1
2
why did throughput drop after update: early impressions - deltapoint17 - 10-15-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
Just sharing a setup that ended up working reasonably well: FLUX, Ollama, 7900 XTX, with a lot of attention on long context.
Maybe useful as a starting point for newer folks.
Re: why did throughput drop after update: early impressions - amberrunner - 10-15-2026
What exact setup are you running? What model, what runtime, and roughly what context size?
If you post your exact settings, people here can probably narrow it down pretty fast.
Re: why did throughput drop after update: early impressions - cacheline - 10-15-2026
I think the bottleneck is somewhere else. I've seen people blame tool calling when the real issue was much more mundane. Could be a workload difference, but I wouldn't generalize too fast from one setup.
Re: why did throughput drop after update: early impressions - turboarch95 - 10-16-2026
A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.
Re: why did throughput drop after update: early impressions - deltapoint17 - 10-16-2026
Appreciate the detail. The pattern around KV cache pressure has been pretty similar for me.
Re: why did throughput drop after update: early impressions - cacheline - 10-16-2026
My first guess would be this: If you're on MLX, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.
Re: why did throughput drop after update: early impressions - amberrunner - 10-16-2026
Quote:My first guess would be this: If you're on MLX, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know t
Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. I expected the model choice to matter more than it did.
Re: why did throughput drop after update: early impressions - routerunner - 10-16-2026
subscribed, curious where this lands.
Re: why did throughput drop after update: early impressions - cacheline - 10-17-2026
Ran a quick check on my box. On mini PC with vLLM, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.
Re: why did throughput drop after update: early impressions - ggufvault - 10-18-2026
Need a little more detail here. What model, what runtime, and roughly what context size?