synthetic data for support bots on 7900 XTX

[phaseport36]

[quote][quote]I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and n

Thanks, that helps. The pattern around batch size has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

I'm happy to test one more variant if you want a second data point.

[amberhall]

that fixed it for me too.

[routecraft]

Not sure I agree there. I get why that seems right, but on A6000 the bottleneck felt elsewhere. Worth testing the simpler explanation before adopting the more interesting one. Turns out 'just one more test' was a lie.

[audioforge]

Not sure I agree there. I get why that seems right, but on A6000 the bottleneck felt elsewhere. Worth testing the simpler explanation before adopting

Thanks, that helps. The pattern around continuous batching has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[circuitworks90]

I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.

Not sure I agree there. I get why that seems right, but on 7900 XTX the bottleneck felt elsewhere. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[quantsupply]

[quote]I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there.