+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: LLM Models (https://www.datasculpting.ai/inference/forumdisplay.php?fid=12)
+--- Forum: Proprietary / Frontier (https://www.datasculpting.ai/inference/forumdisplay.php?fid=48)
+--- Thread: 70B still worth the pain: what are you seeing? (/showthread.php?tid=938)
70B still worth the pain: what are you seeing? - thetaember38 - 10-09-2026
Posting this in case it saves someone else a little time.
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about consumer GPUs.
I'm especially interested in mistakes worth avoiding. The logs were somehow calmer than I was.
Re: 70B still worth the pain: what are you seeing? - routerunner - 10-09-2026
What worked for me was pretty simple. If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.
Re: 70B still worth the pain: what are you seeing? - fastscope - 10-09-2026
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at reranking. Once you know the plain path works, then add the faster path back in piece by piece.
If you post your exact settings, people here can probably narrow it down pretty fast.
Re: 70B still worth the pain: what are you seeing? - latentscope35 - 10-09-2026
A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Phi and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: 70B still worth the pain: what are you seeing? - thetanest - 10-10-2026
subscribed, curious where this lands.
Re: 70B still worth the pain: what are you seeing? - macronest - 10-10-2026
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at speculative decoding. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: 70B still worth the pain: what are you seeing? - cozyarch - 10-10-2026
I think the bottleneck is somewhere else. I've seen people blame tail latency when the real issue was much more mundane. Worth testing the simpler explanation before adopting the more interesting one.