why did throughput drop after update: early impressions

[deltapoint17]

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: FLUX, Ollama, 7900 XTX, with a lot of attention on long context.

Maybe useful as a starting point for newer folks.

[amberrunner]

What exact setup are you running? What model, what runtime, and roughly what context size?

If you post your exact settings, people here can probably narrow it down pretty fast.

[cacheline]

I think the bottleneck is somewhere else. I've seen people blame tool calling when the real issue was much more mundane. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[turboarch95]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.

[deltapoint17]

Appreciate the detail. The pattern around KV cache pressure has been pretty similar for me.

[cacheline]

My first guess would be this: If you're on MLX, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.

[amberrunner]

My first guess would be this: If you're on MLX, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know t

Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. I expected the model choice to matter more than it did.

[routerunner]

subscribed, curious where this lands.

[cacheline]

Ran a quick check on my box. On mini PC with vLLM, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.

[ggufvault]

Need a little more detail here. What model, what runtime, and roughly what context size?