HIP kernel compile issue

[rocmriley]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at VRAM headroom. Once y

I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[quantshed]

Can you fill in a bit more context? What model, what runtime, and roughly what context size?

[rocmmixer]

I'd start with the simpler stack.

[thetatinker77]

That matches what I was seeing. The pattern around embeddings has been pretty similar for me.

[thetashed24]

I don't think that's the whole story. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.