best setup on 24GB VRAM on M3 Max

[torchsmith25]

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on 5090, or squeeze more out of the stack and deal with the complexity.

Trying to avoid rebuilding the whole thing twice.

[southpatch]

Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.

[amberplane]

Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more

Not sure I agree there. I've seen people blame tokens/sec when the real issue was much more mundane.

[emberrunner]

following this because I'm in the same boat.

[cozyarch]

Can you fill in a bit more context? What model, what runtime, and roughly what context size?

[latentcraft]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with FLUX and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.

[cozyarch]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with FLUX and compare from there. Not glamorous advice, but r

Maybe, but I had the opposite result. I've seen people blame consumer GPUs when the real issue was much more mundane. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[kernelmike]

[quote]A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with FLUX and compare from there. Not glamorous advice

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tool calling. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.