+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: LLM Models (https://www.datasculpting.ai/inference/forumdisplay.php?fid=12)
+--- Forum: Proprietary / Frontier (https://www.datasculpting.ai/inference/forumdisplay.php?fid=48)
+--- Thread: 70B still worth the pain (/showthread.php?tid=845)
Pages:
1
2
70B still worth the pain - cobaltworks25 - 10-06-2026
This keeps coming up in conversation so I figured I'd start a thread.
The more I work with local setups, the more it seems like structured output decides everything downstream.
Maybe obvious to some of you, but it clicked for me this week.
Re: 70B still worth the pain - sanevault - 10-06-2026
Ran a quick check on my box. On 7900 XTX with SGLang, I saw basically the same behavior. The main thing that moved was latency, not quality.
Re: 70B still worth the pain - latentscope35 - 10-06-2026
That matches what I was seeing. That matches what I saw on B200. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: 70B still worth the pain - kernelghost - 10-06-2026
Good point. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: 70B still worth the pain - phasetrack - 10-06-2026
Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet. At this point the real bottleneck is my patience.
Re: 70B still worth the pain - mellowstream - 10-07-2026
A couple questions before I guess too hard. Are you testing this on B200 or something else?
If you post your exact settings, people here can probably narrow it down pretty fast.
Re: 70B still worth the pain - cacheline - 10-08-2026
Quote:A couple questions before I guess too hard. Are you testing this on B200 or something else?
Good point. That matches what I saw on dual 3090. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: 70B still worth the pain - tensorriver - 10-08-2026
Quote:Quote:A couple questions before I guess too hard. Are you testing this on B200 or something else?
Good point. That matches what I saw on 7900 XTX.
Re: 70B still worth the pain - macronest - 10-08-2026
I'd start with the simpler stack.
Re: 70B still worth the pain - macrolab - 10-08-2026
Quote:I'd start with the simpler stack.
that fixed it for me too.