serving on Blackwell for FLUX

[kernelmike]

Did a bit of testing and thought I'd post the result here.

Short version: LM Studio behaved well for me, but only after I stopped ignoring VRAM headroom.

Nothing scientific here, just practical notes from a real setup.

[audioforge]

Thanks, that helps. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

I'm happy to test one more variant if you want a second data point.

[pixelcrate82]

Thanks, that helps. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimizat

Maybe, but I had the opposite result. I've seen people blame batch size when the real issue was much more mundane. Could be a workload difference, but I wouldn't generalize too fast from one setup. The logs were somehow calmer than I was.

[kernelmike]

Need a little more detail here. Do you see the same thing with a smaller batch or shorter prompt?

[vectorport]

Need a little more detail here. What model, what runtime, and roughly what context size?

[sysstation69]

A couple things I'd check first. If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.

I'm happy to test one more variant if you want a second data point.

[edgecoast]

My first guess would be this: On 4090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[cacheline]

I don't think that's the whole story. I've seen people blame long context when the real issue was much more mundane. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[circuitkite]

Thanks, that helps. I don't think your conclusion is far off.

[oxidegrove40]

Thanks, that helps. I don't think your conclusion is far off.

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at router policy. Not glamorous advice, but repeatable baselines save a lot of time.