+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Harnesses, Agents & Orchestration (https://www.datasculpting.ai/inference/forumdisplay.php?fid=11)
+--- Forum: Workflows / Automation (https://www.datasculpting.ai/inference/forumdisplay.php?fid=47)
+--- Thread: which hosted model is actually worth paying for (/showthread.php?tid=1327)
Pages:
1
2
which hosted model is actually worth paying for - routecraft - 10-16-2026
Did a bit of testing and thought I'd post the result here.
I tried this with Phi using LM Studio on B200. The headline result was better than I expected, but batch size still ended up being the limiting factor.
Curious if anyone else landed in the same place.
I care more about stability than squeezing the last possible tokens/sec out of it.
Re: which hosted model is actually worth paying for - routecraft - 10-16-2026
A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: which hosted model is actually worth paying for - cobaltworks25 - 10-16-2026
Thanks, that helps. That matches what I saw on GB200. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: which hosted model is actually worth paying for - routecraft - 10-17-2026
My first guess would be this: If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: which hosted model is actually worth paying for - nanogarden - 10-18-2026
I don't think that's the whole story. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.
Re: which hosted model is actually worth paying for - routeline23 - 10-18-2026
Quote:That matches what I was seeing. I ended up in about the same place using llama.cpp. Worth rechecking your defaults though, because one bad assumption
subscribed, curious where this lands.
Re: which hosted model is actually worth paying for - dockworks23 - 10-18-2026
Not sure I agree there. I get why that seems right, but on M3 Max the bottleneck felt elsewhere. Worth testing the simpler explanation before adopting the more interesting one.
Re: which hosted model is actually worth paying for - routecraft - 10-19-2026
A couple things I'd check first. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.
Re: which hosted model is actually worth paying for - routecraft - 10-19-2026
I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at quant drift. Once you know the plain path works, then add the faster path back in piece by piece.
Re: which hosted model is actually worth paying for - phasestream53 - 10-19-2026
I'd start with the simpler stack.