+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: llama.cpp / GGUF (https://www.datasculpting.ai/inference/forumdisplay.php?fid=35)
+--- Thread: multi-GPU offload advice (/showthread.php?tid=73)
Pages:
1
2
multi-GPU offload advice - bytesupply76 - 08-31-2026
Posting this in case it saves someone else a little time.
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about token cost.
Would rather learn from your scars than collect my own all at once. At this point the real bottleneck is my patience.
Re: multi-GPU offload advice - dockridge - 08-31-2026
Not sure I agree there. I get why that seems right, but on mini PC the bottleneck felt elsewhere.
Re: multi-GPU offload advice - phasescope - 08-31-2026
following this because I'm in the same boat.
Re: multi-GPU offload advice - thetagrid15 - 08-31-2026
What worked for me was pretty simple. If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.
Re: multi-GPU offload advice - macroforge - 08-31-2026
Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.
Re: multi-GPU offload advice - bytesupply76 - 08-31-2026
I hit almost the same issue last night.
Re: multi-GPU offload advice - llm_wren - 08-31-2026
that fixed it for me too.
Re: multi-GPU offload advice - ggufwire22 - 08-31-2026
Quote:that fixed it for me too.
Good point. The pattern around workstation thermals has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: multi-GPU offload advice - sparrowkite53 - 08-31-2026
I'd approach it in a boring order. If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious.
Re: multi-GPU offload advice - embershed99 - 09-01-2026
Quote:I'd approach it in a boring order. If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious.
Interesting, I hadn't tried that. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.