domain tune without wrecking base behavior

[routearch]

Did a bit of testing and thought I'd post the result here.

I tried this with DeepSeek using vLLM on M3 Max. The headline result was better than I expected, but tokens/sec still ended up being the limiting factor.

Curious if anyone else landed in the same place.

[kernelarch]

My first guess would be this: On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[quiethouse]

My first guess would be this: On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives.

Ran a quick check on my box. On 4090 with llama.cpp, I saw basically the same behavior. I expected the model choice to matter more than it did.

[deltastation]

Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher.

[batchfield55]

I don't think that's the whole story. I get why that seems right, but on GB200 the bottleneck felt elsewhere. Could be a workload difference, but I wouldn't generalize too fast from one setup.

If you post your exact settings, people here can probably narrow it down pretty fast.

[signalcraft14]

that fixed it for me too.

[quiethouse]

Not sure I agree there. I get why that seems right, but on dual 3090 the bottleneck felt elsewhere. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[localtinker]

Not sure I agree there. I get why that seems right, but on dual 3090 the bottleneck felt elsewhere. Not saying you're wrong, just that I wouldn't lock

Thanks, that helps. I ended up in about the same place using SGLang. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[routearch]

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Phi and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.

[cacheline]

What exact setup are you running? Are you testing this on B200 or something else?