serving on Blackwell

[batchcraft]

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: PyTorch on Threadripper box, keeping prefix caching in mind from the start.

Maybe useful as a starting point for newer folks.

I care more about stability than squeezing the last possible tokens/sec out of it. At this point the real bottleneck is my patience.

[pixelcrate82]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at consumer GPUs. Not glamorous advice, but repeatable baselines save a lot of time.

[rocmstack25]

That matches what I was seeing. The pattern around long context has been pretty similar for me.

[sysstation69]

I don't think that's the whole story. I've seen people blame tool calling when the real issue was much more mundane. Could be a workload difference, but I wouldn't generalize too fast from one setup.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[pixelcrate82]

I'd start with the simpler stack.

[vectorport]

Thanks, that helps. The pattern around token cost has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[thetarunner]

What worked for me was pretty simple. On B200, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lo

A couple things I'd check first. On GB200, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

If you post your exact settings, people here can probably narrow it down pretty fast.

[pixelcrate82]

I hit almost the same issue last night.

[rasterbench]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Phi and compare from there.

[visionyard23]

Interesting, I hadn't tried that. The pattern around tool calling has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.