multi-GPU offload advice

[bytesupply76]

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about token cost.

Would rather learn from your scars than collect my own all at once. At this point the real bottleneck is my patience.

[dockridge]

Not sure I agree there. I get why that seems right, but on mini PC the bottleneck felt elsewhere.

[phasescope]

following this because I'm in the same boat.

[thetagrid15]

What worked for me was pretty simple. If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.

[macroforge]

Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[bytesupply76]

I hit almost the same issue last night.

[llm_wren]

that fixed it for me too.

[ggufwire22]

that fixed it for me too.

Good point. The pattern around workstation thermals has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[sparrowkite53]

I'd approach it in a boring order. If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious.

[embershed99]

I'd approach it in a boring order. If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious.

Interesting, I hadn't tried that. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.