where reranking mattered for FLUX

[labpath]

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of PyTorch and 4090 felt surprisingly usable once I tuned around KV cache pressure.

Maybe useful as a starting point for newer folks.

[quietcraft]

What worked for me was pretty simple. On M3 Max, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

[audionode8]

A couple questions before I guess too hard. Is this local-only or are you routing anything externally?

[sparrowbrook]

What exact setup are you running? Is this local-only or are you routing anything externally?

[routecrate]

Maybe, but I had the opposite result. I get why that seems right, but on GB200 the bottleneck felt elsewhere.

[sanepeak48]

Tried to reproduce it just now. On 4090 with MLX, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.

[macropilot]

A couple questions before I guess too hard. What model, what runtime, and roughly what context size?

[visionforge]

I hit almost the same issue last night.

[tensorfield88]

I hit almost the same issue last night.

Maybe, but I had the opposite result. I've seen people blame router policy when the real issue was much more mundane.

[kernelghost]

I hit almost the same issue last night.

I'd start with the simpler stack.