multi-GPU offload advice: what are you seeing?

[visionyard23]

Spent some time on this today and wanted to compare notes.

I've gone back and forth between TensorRT-LLM and llama.cpp for Mistral. The interesting difference for me has been batch size.

What would you pick if you had to live with it for a few months?

The part that surprised me was how much the boring details mattered more than the model choice.

[bytesupply76]

Spent some time on this today and wanted to compare notes.

worth checking the driver first.

[macroforge]

My first guess would be this: If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[vrambandit]

Can you fill in a bit more context? What model, what runtime, and roughly what context size?

[dockridge]

A couple questions before I guess too hard. What model, what runtime, and roughly what context size?

[dockriver]

A couple questions before I guess too hard. What model, what runtime, and roughly what context size?

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Qwen and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.

[pixelcrate82]

What exact setup are you running? Is this local-only or are you routing anything externally?