7900 XTX local stack: worth it?

[phasetrack]

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about small local stacks.

Simple and boring is a valid answer.

[papermixer4]

Posting this in case it saves someone else a little time.

Good point. I ended up in about the same place using Ollama.

[dockworks23]

I tested this tonight. On Threadripper box with PyTorch, I saw basically the same behavior. The main thing that moved was latency, not quality.

[relaynest82]

I tested this tonight. On Threadripper box with PyTorch, I saw basically the same behavior. The main thing that moved was latency, not quality.

Good point. The pattern around tokens/sec has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[rocmpath]

Appreciate the detail. That matches what I saw on Threadripper box. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[papermixer4]

My first guess would be this: If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious.