how strict do you make schemas

[northcraft90]

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like structured output decides everything downstream.

Wondering if this is just where the whole space is heading now.

[signalcraft14]

This keeps coming up in conversation so I figured I'd start a thread.

Tried to reproduce it just now. On GB200 with llama.cpp, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.

[audioforge]

A couple things I'd check first. On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[llmarch51]

Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly.

[fusedhall]

Interesting, I hadn't tried that. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[llmtrack26]

I'd start with the simpler stack.

[torchdesk]

same here on my side.

[kernelhouse70]

Good point. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[mintforge]

Good point. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow f

same here on my side.

[cacheline]

Can you fill in a bit more context? Are you testing this on B200 or something else?