MCP server discovery mess with vLLM

[quiettoken]

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on GB200, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?

This is mostly for local work rather than public serving.

[mangofield33]

Spent some time on this today and wanted to compare notes.

Ran a quick check on my box. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[plainworks]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at MoE serving.

[pixelcrate82]

Thanks, that helps. The pattern around tail latency has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[mangowire]

Not sure I agree there. I've seen people blame token cost when the real issue was much more mundane.

[mellowstream]

Not sure I agree there. I've seen people blame token cost when the real issue was much more mundane.

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tail latency. Not glamorous advice, but repeatable baselines save a lot of time.

[torchshed66]

Not sure I agree there. I've seen people blame token cost when the real issue was much more mundane.

I tested this tonight. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.

[bytepilot]

I think the bottleneck is somewhere else. I get why that seems right, but on mini PC the bottleneck felt elsewhere. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[labnest]

Can you fill in a bit more context? Are you testing this on GB200 or something else?

If you post your exact settings, people here can probably narrow it down pretty fast.

[ggufwire22]

Good point. I ended up in about the same place using Ollama. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.