VRAM saving for image generation for FLUX

[rocmrunner40]

Spent some time on this today and wanted to compare notes.

I've gone back and forth between LM Studio and SGLang for Llama. The interesting difference for me has been NVFP4.

Trying to avoid rebuilding the whole thing twice.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

[nanopatch]

That matches what I was seeing. The pattern around VRAM headroom has been pretty similar for me.

[vectorpoint]

A couple questions before I guess too hard. What model, what runtime, and roughly what context size?

[fastbench]

What exact setup are you running? Is this local-only or are you routing anything externally?

[thetapath]

What exact setup are you running? Is this local-only or are you routing anything externally?

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with FLUX and compare from there.

[turboarch95]

I'd start with the simpler stack.

[batchfield55]

I'd start with the simpler stack.

Interesting, I hadn't tried that. The pattern around continuous batching has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

I'm happy to test one more variant if you want a second data point.

[kernelrunner]

Good point. That matches what I saw on dual 3090. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[thetarunner]

What exact setup are you running? Is this local-only or are you routing anything externally?

I'm happy to test one more variant if you want a second data point.

[cozyarch]

Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. I expected the model choice to matter more than it did.