which hosted model is actually worth paying for

[routecraft]

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like workstation thermals decides everything downstream.

Maybe obvious to some of you, but it clicked for me this week.

This is mostly for local work rather than public serving.

[routecraft]

A couple things I'd check first. If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[flintriver50]

I hit almost the same issue last night.

[routecraft]

Need a little more detail here. Are you testing this on 7900 XTX or something else?

[mangoquill]

That matches what I was seeing. I ended up in about the same place using vLLM. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[routecraft]

Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.

[amberhall]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tool calling. Not glamorous advice, but repeatable baselines save a lot of time.

[routecraft]

Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.

[relaynest82]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at KV cache pressure. Once you know the plain path works, then add the faster path back in piece by piece.

[routecraft]

I don't think that's the whole story. I get why that seems right, but on 4090 the bottleneck felt elsewhere. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.