VRAM saving for image generation for Gemma

[mangopeak72]

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on M3 Max, or squeeze more out of the stack and deal with the complexity.

Interested in tradeoffs more than fan loyalty.

[circuitwire58]

I tested this tonight. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.

[vectorport]

Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.

[cobalttinker]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at structured output. Once you know the plain path works, then add the faster path back in piece by piece.

[torchtrack24]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there.

[orbitline]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at speculative decoding. Not glamorous advice, but repeatable baselines save a lot of time.

[stackpilot]

I hit almost the same issue last night.

[mangoquill]

That matches what I was seeing. The pattern around batch size has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[llmarch83]

Maybe, but I had the opposite result. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[mangovault67]

That matches what I was seeing. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.