desk setup thread for Gemma

[bytegarden73]

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on 4090, or squeeze more out of the stack and deal with the complexity.

Trying to avoid rebuilding the whole thing twice.

I care more about stability than squeezing the last possible tokens/sec out of it.

[kernelmike]

Need a little more detail here. Are you testing this on 7900 XTX or something else?

[tinycontext]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at workstation thermals. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[benchplease]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at workstation thermals. A lot o

Thanks, that helps. I don't think your conclusion is far off.

[kernelarch]

Interesting, I hadn't tried that. I ended up in about the same place using LM Studio.

[nanopatch]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at workstation thermals.

[vectorport]

Maybe, but I had the opposite result. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the simpler explanation before adopting the more interesting one.

[batchshed]

I'd approach it in a boring order. On 7900 XTX, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[signaltrack44]

I'd approach it in a boring order. On 7900 XTX, the easiest win is often to back off the aggressive settings and see whether the problem survives.

I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.

[labnest]

I tested this tonight. On 7900 XTX with LM Studio, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.