dual GPU split inference with LM Studio

[signaltrack44]

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of Ollama and dual 3090 felt surprisingly usable once I tuned around quant drift.

Maybe useful as a starting point for newer folks.

This is mostly for local work rather than public serving.

[sparrowgrid]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Llama and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[tensorfield88]

What exact setup are you running? Do you see the same thing with a smaller batch or shorter prompt?

[sanevault]

That matches what I was seeing. I ended up in about the same place using MLX. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[routeforge64]

A couple things I'd check first. If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious.

[paperrunner]

I tested this tonight. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.