+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: TensorRT-LLM (https://www.datasculpting.ai/inference/forumdisplay.php?fid=37)
+--- Thread: serving on Blackwell (/showthread.php?tid=1354)
Pages:
1
2
serving on Blackwell - batchcraft - 10-16-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
Maybe useful as a starting point: PyTorch on Threadripper box, keeping prefix caching in mind from the start.
Maybe useful as a starting point for newer folks.
I care more about stability than squeezing the last possible tokens/sec out of it. At this point the real bottleneck is my patience.
Re: serving on Blackwell - pixelcrate82 - 10-17-2026
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at consumer GPUs. Not glamorous advice, but repeatable baselines save a lot of time.
Re: serving on Blackwell - rocmstack25 - 10-17-2026
That matches what I was seeing. The pattern around long context has been pretty similar for me.
Re: serving on Blackwell - sysstation69 - 10-17-2026
I don't think that's the whole story. I've seen people blame tool calling when the real issue was much more mundane. Could be a workload difference, but I wouldn't generalize too fast from one setup.
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.
Re: serving on Blackwell - pixelcrate82 - 10-17-2026
I'd start with the simpler stack.
Re: serving on Blackwell - vectorport - 10-18-2026
Thanks, that helps. The pattern around token cost has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: serving on Blackwell - thetarunner - 10-19-2026
Quote:What worked for me was pretty simple. On B200, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lo
A couple things I'd check first. On GB200, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
If you post your exact settings, people here can probably narrow it down pretty fast.
Re: serving on Blackwell - pixelcrate82 - 10-19-2026
I hit almost the same issue last night.
Re: serving on Blackwell - rasterbench - 10-19-2026
I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Phi and compare from there.
Re: serving on Blackwell - visionyard23 - 10-19-2026
Interesting, I hadn't tried that. The pattern around tool calling has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.