small win thread

[quiettoken]

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of llama.cpp and 7900 XTX felt surprisingly usable once I tuned around prefill/decode split.

Feel free to tear it apart if you see something dumb.

[quiettoken]

Good point. That matches what I saw on 4090. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[rocmrunner40]

Interesting, I hadn't tried that. The pattern around speculative decoding has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[cacheplane98]

that fixed it for me too.

[thetagrid15]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefill/decode split. Not glamorous advice, but repeatable baselines save a lot of time.

[rocmrunner40]

Maybe, but I had the opposite result. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Worth testing the simpler explanation before adopting the more interesting one.

[relaysupply90]

A couple things I'd check first. On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

[rocmplane]

That matches what I was seeing. I ended up in about the same place using PyTorch. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[sparrowgrid]

Thanks, that helps. That matches what I saw on A6000. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[vectorshed43]

What worked for me was pretty simple. On B200, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.