routing between local and hosted: what are you seeing?

[cobaltworks25]

Has anyone else run into this?

My current setup is TensorRT-LLM with Threadripper box, and I keep running into VRAM headroom.

Not looking for the perfect answer, mostly what worked in real use.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.

[northgrid33]

Good point. The pattern around structured output has been pretty similar for me.

[sparrowkite53]

Interesting, I hadn't tried that. That matches what I saw on 5090.

[signalpeak]

Interesting, I hadn't tried that. I ended up in about the same place using TensorRT-LLM. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

If you post your exact settings, people here can probably narrow it down pretty fast.

[northhall88]

Not sure I agree there. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.

[turboscope5]

What worked for me was pretty simple. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.

[edgequill]

My first guess would be this: On M3 Max, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[circuitkite]

Need a little more detail here. Is this local-only or are you routing anything externally?

[cudahouse]

Need a little more detail here. Is this local-only or are you routing anything externally?

A couple things I'd check first. If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.