structured output failures

[homelabhenry]

Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.

[thetarunner]

subscribed, curious where this lands.

If you post your exact settings, people here can probably narrow it down pretty fast.

[bytesupply76]

Good point. The pattern around prefill/decode split has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[routecrate]

Good point. The pattern around prefill/decode split has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption

Good point. The pattern around workstation thermals has been pretty similar for me. Turns out 'just one more test' was a lie.

[cudaline]

[quote]Good point. The pattern around prefill/decode split has been pretty similar for me. Worth rechecking your defaults though, because one bad assu

Maybe, but I had the opposite result. I've seen people blame workstation thermals when the real issue was much more mundane.

[cachelab]

That matches what I was seeing. I ended up in about the same place using llama.cpp.