+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: LLM Models (https://www.datasculpting.ai/inference/forumdisplay.php?fid=12)
+--- Forum: Proprietary / Frontier (https://www.datasculpting.ai/inference/forumdisplay.php?fid=48)
+--- Thread: 70B still worth the pain (/showthread.php?tid=165)
Pages:
1
2
70B still worth the pain - rocmhouse90 - 09-06-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
Just sharing a setup that ended up working reasonably well: Phi, SGLang, B200, with a lot of attention on prefix caching.
Feel free to tear it apart if you see something dumb.
If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.
Re: 70B still worth the pain - cudascope - 09-06-2026
What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at INT4. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: 70B still worth the pain - mangotinker73 - 09-07-2026
Thanks, that helps. That matches what I saw on M3 Max. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.
Re: 70B still worth the pain - mangoquill - 09-07-2026
Quote:Thanks, that helps. That matches what I saw on M3 Max. I got better results once I stopped chasing one metric and just looked at whether the workflow
I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at router policy. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: 70B still worth the pain - quiethouse - 09-10-2026
I think the bottleneck is somewhere else. I've seen people blame small local stacks when the real issue was much more mundane. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.
Re: 70B still worth the pain - latentpeak22 - 09-10-2026
Quote:I think the bottleneck is somewhere else. I've seen people blame small local stacks when the real issue was much more mundane. Not saying you're wrong
I'd start with the simpler stack.
Re: 70B still worth the pain - bytehall - 09-13-2026
Quote:[quote]I think the bottleneck is somewhere else. I've seen people blame small local stacks when the real issue was much more mundane. Not saying you'r
That matches what I was seeing. I ended up in about the same place using MLX. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: 70B still worth the pain - turboyard94 - 09-13-2026
Did a small comparison this afternoon. On GB200 with SGLang, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.
Re: 70B still worth the pain - batchscope72 - 09-15-2026
I hit almost the same issue last night.
Re: 70B still worth the pain - cudaline - 09-15-2026
Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.