+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: llama.cpp / GGUF (https://www.datasculpting.ai/inference/forumdisplay.php?fid=35)
+--- Thread: multi-GPU offload advice: what are you seeing? (/showthread.php?tid=836)
multi-GPU offload advice: what are you seeing? - visionyard23 - 10-07-2026
Spent some time on this today and wanted to compare notes.
I've gone back and forth between TensorRT-LLM and llama.cpp for Mistral. The interesting difference for me has been batch size.
What would you pick if you had to live with it for a few months?
The part that surprised me was how much the boring details mattered more than the model choice.
Re: multi-GPU offload advice: what are you seeing? - bytesupply76 - 10-07-2026
Quote:Spent some time on this today and wanted to compare notes.
worth checking the driver first.
Re: multi-GPU offload advice: what are you seeing? - macroforge - 10-09-2026
My first guess would be this: If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: multi-GPU offload advice: what are you seeing? - vrambandit - 10-09-2026
Can you fill in a bit more context? What model, what runtime, and roughly what context size?
Re: multi-GPU offload advice: what are you seeing? - dockridge - 10-09-2026
A couple questions before I guess too hard. What model, what runtime, and roughly what context size?
Re: multi-GPU offload advice: what are you seeing? - dockriver - 10-09-2026
Quote:A couple questions before I guess too hard. What model, what runtime, and roughly what context size?
My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Qwen and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.
Re: multi-GPU offload advice: what are you seeing? - pixelcrate82 - 10-09-2026
What exact setup are you running? Is this local-only or are you routing anything externally?