RTX 5090 local inference for FLUX

[flintriver50]

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of llama.cpp and 7900 XTX felt surprisingly usable once I tuned around token cost.

Maybe useful as a starting point for newer folks.

[turbogrid11]

Thought I'd post a small recipe because a lot of threads ask similar things.

My first guess would be this: If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious.

[rocmriley]

A couple things I'd check first. If you're on Ollama, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[torchshed]

Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.

[rocmriley]

A couple things I'd check first. If you're on Ollama, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.

[signaltrack44]

Interesting, I hadn't tried that. The pattern around batch size has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[turbogrid11]

Ran a quick check on my box. The issue went away once I changed one setting and restarted the stack cleanly.

[fusedhall]

What exact setup are you running? Do you see the same thing with a smaller batch or shorter prompt?

[vrambandit]

What exact setup are you running? Is this local-only or are you routing anything externally?

[cudapoint]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tokens/sec. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.