70B still worth the pain

[cobaltworks25]

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like structured output decides everything downstream.

Maybe obvious to some of you, but it clicked for me this week.

[sanevault]

Ran a quick check on my box. On 7900 XTX with SGLang, I saw basically the same behavior. The main thing that moved was latency, not quality.

[latentscope35]

That matches what I was seeing. That matches what I saw on B200. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[kernelghost]

Good point. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[phasetrack]

Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet. At this point the real bottleneck is my patience.

[mellowstream]

A couple questions before I guess too hard. Are you testing this on B200 or something else?

If you post your exact settings, people here can probably narrow it down pretty fast.

[cacheline]

A couple questions before I guess too hard. Are you testing this on B200 or something else?

Good point. That matches what I saw on dual 3090. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[tensorriver]

A couple questions before I guess too hard. Are you testing this on B200 or something else?

Good point. That matches what I saw on 7900 XTX.

[macronest]

I'd start with the simpler stack.

[macrolab]

I'd start with the simpler stack.

that fixed it for me too.