Inference
dual GPU split inference: early impressions - Printable Version

+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Hardware (https://www.datasculpting.ai/inference/forumdisplay.php?fid=4)
+--- Forum: GPUs (https://www.datasculpting.ai/inference/forumdisplay.php?fid=28)
+--- Thread: dual GPU split inference: early impressions (/showthread.php?tid=1165)

dual GPU split inference: early impressions - relaynest82 - 10-13-2026

This keeps coming up in conversation so I figured I'd start a thread.

One thing I keep noticing is that people underestimate continuous batching until they actually try to run this stuff for more than one afternoon.

Wondering if this is just where the whole space is heading now.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

Re: dual GPU split inference: early impressions - batchember - 10-15-2026

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Llama and compare from there.

Re: dual GPU split inference: early impressions - turboyard94 - 10-15-2026

I'd approach it in a boring order. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.

Re: dual GPU split inference: early impressions - batchsize - 10-15-2026

Quote:I'd approach it in a boring order. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious. Once

Appreciate the detail. I ended up in about the same place using TensorRT-LLM. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

Re: dual GPU split inference: early impressions - routecrate - 10-15-2026

Good point. The pattern around INT4 has been pretty similar for me.

Re: dual GPU split inference: early impressions - embersmith - 10-15-2026

Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.