system prompt habits that helped for Mixtral

[quiettoken]

Thanks, that helps. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[thetaarch43]

Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?

[cudapeak]

Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?

Interesting, I hadn't tried that. That matches what I saw on 4090.

[thetafield]

Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed. At this point the real bottleneck is my patience.