structured output failures with PyTorch

[cozygrid12]

Did a bit of testing and thought I'd post the result here.

After a few runs on 5090, I ended up with decent performance, but the setup only really made sense once I adjusted around long context.

Happy to post more details if anyone wants them.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

[quietsupply11]

Did a bit of testing and thought I'd post the result here.

My first guess would be this: On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.

I'm happy to test one more variant if you want a second data point.

[kernelmike]

Did a bit of testing and thought I'd post the result here.

I hit almost the same issue last night.

[benchplease]

worth checking the driver first.

[quantshed]

same here on my side.

[oxiderunner]

I'd start with the simpler stack.

[rocmpath]

What exact setup are you running? Is this local-only or are you routing anything externally?

[ggufwire22]

Ran a quick check on my box. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.

[audiodesk]

Interesting, I hadn't tried that. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[localcraft17]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there.