comparing models without fooling yourself on GB200

[deltastation]

I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly.

[griffinloom66]

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of vLLM and dual 3090 felt surprisingly usable once I tuned around tool calling.

Feel free to tear it apart if you see something dumb.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

[llmtrack26]

What worked for me was pretty simple. On A6000, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.

[kernelghost]

I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher. I expected the model choice to matter more than it did.

[plainember52]

Interesting, I hadn't tried that. I ended up in about the same place using SGLang. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[vramridge]

I'd start with the simpler stack.

[quiethouse]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at continuous batching. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[circuitworks90]

same here on my side.

[batchpeak]

That matches what I was seeing. I ended up in about the same place using SGLang. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[deltastation]

I'd approach it in a boring order. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious.