ROCm version that finally behaves for Gemma

[benchplease]

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Phi, LM Studio, 5090, with a lot of attention on long context.

If someone wants the exact settings I can clean them up and post them.

[audiopath66]

Thought I'd post a small recipe because a lot of threads ask similar things.

I'd approach it in a boring order. On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.

[dockworks23]

Maybe, but I had the opposite result. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[torchsmith25]

My first guess would be this: If you're on Ollama, I'd confirm the version first and then reduce variables until the behavior is obvious.

[southpatch]

I hit almost the same issue last night.

[rocmriley]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at INT4. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[torchsmith25]

I don't think that's the whole story. I get why that seems right, but on B200 the bottleneck felt elsewhere.

What worked for me was pretty simple. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious.

[northgrid33]

subscribed, curious where this lands.

[papermixer4]

Good point. The pattern around VRAM headroom has been pretty similar for me.

[quantshed]

I don't think that's the whole story. I get why that seems right, but on B200 the bottleneck felt elsewhere.