+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Harnesses, Agents & Orchestration (https://www.datasculpting.ai/inference/forumdisplay.php?fid=11)
+--- Forum: MCP / APIs / Tool Calling (https://www.datasculpting.ai/inference/forumdisplay.php?fid=46)
+--- Thread: model merge got better or just different (/showthread.php?tid=243)
Pages:
1
2
model merge got better or just different - mangowire - 09-12-2026
Did a bit of testing and thought I'd post the result here.
I tried this with Phi using llama.cpp on A6000. The headline result was better than I expected, but VRAM headroom still ended up being the limiting factor.
Happy to post more details if anyone wants them.
Re: model merge got better or just different - phaseshed - 09-12-2026
My first guess would be this: If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious.
Re: model merge got better or just different - circuitworks90 - 09-13-2026
Need a little more detail here. Are you testing this on mini PC or something else? I keep promising myself this is the last rebuild.
Re: model merge got better or just different - routerunner - 09-13-2026
What exact setup are you running? Do you see the same thing with a smaller batch or shorter prompt?
Re: model merge got better or just different - batchscope72 - 09-13-2026
Ran a quick check on my box. On 4090 with MLX, I saw basically the same behavior.
Re: model merge got better or just different - batchsize - 09-13-2026
Ran a quick check on my box. On 7900 XTX with MLX, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.
Re: model merge got better or just different - vramstream - 09-15-2026
A couple things I'd check first. On 4090, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Re: model merge got better or just different - griffinloom66 - 09-15-2026
I think the bottleneck is somewhere else. I get why that seems right, but on 4090 the bottleneck felt elsewhere. Could be a workload difference, but I wouldn't generalize too fast from one setup.
Re: model merge got better or just different - tensorport - 09-15-2026
My first guess would be this: On GB200, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Re: model merge got better or just different - localpatch78 - 09-15-2026
Quote:My first guess would be this: On GB200, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. I expected the model choice to matter more than it did.