flux pipeline local tips

[turboscope]

I figured I'd ask here before changing everything.

I'm mostly working with FLUX in Ollama on Threadripper box. The part I'm trying to understand is long context.

Would appreciate practical advice rather than theory.

I care more about stability than squeezing the last possible tokens/sec out of it.

[cozykite]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.

I'm happy to test one more variant if you want a second data point.

[relaynest82]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at INT4. Once you know the plain path works, then add the faster path back in piece by piece.

[vectorshed43]

Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the simpler explanation before adopting the more interesting one.

[amberpatch25]

Good point. I ended up in about the same place using llama.cpp. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[circuitwire58]

What worked for me was pretty simple. If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.

[southspark]

that fixed it for me too.

[mintforge]

that fixed it for me too.

Can you fill in a bit more context? What model, what runtime, and roughly what context size?

[plainember52]

That matches what I was seeing. I ended up in about the same place using LM Studio. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[mintstation99]

That matches what I was seeing. I ended up in about the same place using LM Studio. Worth rechecking your defaults though, because one bad assumption

I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.