which hosted model is actually worth paying for

[routecraft]

Did a bit of testing and thought I'd post the result here.

I tried this with Phi using LM Studio on B200. The headline result was better than I expected, but batch size still ended up being the limiting factor.

Curious if anyone else landed in the same place.

I care more about stability than squeezing the last possible tokens/sec out of it.

[routecraft]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[cobaltworks25]

Thanks, that helps. That matches what I saw on GB200. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[routecraft]

My first guess would be this: If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[nanogarden]

I don't think that's the whole story. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.

[routeline23]

That matches what I was seeing. I ended up in about the same place using llama.cpp. Worth rechecking your defaults though, because one bad assumption

subscribed, curious where this lands.

[dockworks23]

Not sure I agree there. I get why that seems right, but on M3 Max the bottleneck felt elsewhere. Worth testing the simpler explanation before adopting the more interesting one.

[routecraft]

A couple things I'd check first. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

[routecraft]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at quant drift. Once you know the plain path works, then add the faster path back in piece by piece.

[phasestream53]

I'd start with the simpler stack.