is RDNA worth the hassle yet with llama.cpp

[relaynest82]

Spent some time on this today and wanted to compare notes.

I've gone back and forth between vLLM and MLX for Gemma. The interesting difference for me has been HBM constraints.

Trying to avoid rebuilding the whole thing twice.

I care more about stability than squeezing the last possible tokens/sec out of it.

[mintquill]

Need a little more detail here. Do you see the same thing with a smaller batch or shorter prompt?