model merge got better or just different

[mangowire]

Did a bit of testing and thought I'd post the result here.

I tried this with Phi using llama.cpp on A6000. The headline result was better than I expected, but VRAM headroom still ended up being the limiting factor.

Happy to post more details if anyone wants them.

[phaseshed]

My first guess would be this: If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious.

[circuitworks90]

Need a little more detail here. Are you testing this on mini PC or something else? I keep promising myself this is the last rebuild.

[routerunner]

What exact setup are you running? Do you see the same thing with a smaller batch or shorter prompt?

[batchscope72]

Ran a quick check on my box. On 4090 with MLX, I saw basically the same behavior.

[batchsize]

Ran a quick check on my box. On 7900 XTX with MLX, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.

[vramstream]

A couple things I'd check first. On 4090, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[griffinloom66]

I think the bottleneck is somewhere else. I get why that seems right, but on 4090 the bottleneck felt elsewhere. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[tensorport]

My first guess would be this: On GB200, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[localpatch78]

My first guess would be this: On GB200, the easiest win is often to back off the aggressive settings and see whether the problem survives.

Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. I expected the model choice to matter more than it did.