Blackwell deployment chatter for FLUX

[thetapath]

Curious what people here are seeing.

I'm mostly working with Qwen in LM Studio on 4090. The part I'm trying to understand is prefix caching.

If you've tested something similar, what did you keep and what did you stop bothering with?

[edgecoast]

What exact setup are you running? Do you see the same thing with a smaller batch or shorter prompt?

[kernelmike]

Good point. The pattern around MoE serving has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[dockember]

Good point. The pattern around MoE serving has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make

I'd start with the simpler stack.

[audiopath66]

Good point. The pattern around batch size has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[sparrowkite53]

Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.

[cobaltspark]

Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.

I don't think that's the whole story. I've seen people blame tokens/sec when the real issue was much more mundane. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[phaseport36]

I tested this tonight. On Threadripper box with TensorRT-LLM, I saw basically the same behavior. The main thing that moved was latency, not quality.

I'm happy to test one more variant if you want a second data point.

[sparrowkite53]

Not sure I agree there. I've seen people blame consumer GPUs when the real issue was much more mundane.

[phaseport36]

worth checking the driver first.