new here, what's your stack on 5090

[quiettoken]

I'd start with the simpler stack.

[routeline23]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[thetagrid15]

Maybe, but I had the opposite result. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Worth testing the simpler explanation before adopting the more interesting one.

[marbletrack]

Good point. I ended up in about the same place using vLLM. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

If you post your exact settings, people here can probably narrow it down pretty fast.

[quiettoken]

Good point. I ended up in about the same place using vLLM. Worth rechecking your defaults though, because one bad assumption can make the rest of the

I don't think that's the whole story. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the simpler explanation before adopting the more interesting one.

[rocmplane]

[quote]Good point. I ended up in about the same place using vLLM. Worth rechecking your defaults though, because one bad assumption can make the rest

I hit almost the same issue last night.

[cacheline]

Interesting, I hadn't tried that. That matches what I saw on 5090. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[turboarch95]

That matches what I was seeing. That matches what I saw on 7900 XTX. The biggest change for me was simplifying the stack instead of adding one more optimization layer. The logs were somehow calmer than I was.

[rocmrunner40]

Thanks, that helps. That matches what I saw on A6000. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[vectorport]

I don't think that's the whole story. I've seen people blame batch size when the real issue was much more mundane. Could be a workload difference, but I wouldn't generalize too fast from one setup.