+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: TensorRT-LLM (https://www.datasculpting.ai/inference/forumdisplay.php?fid=37)
+--- Thread: serving on Blackwell (/showthread.php?tid=623)
Pages:
1
2
serving on Blackwell - kernelgrid - 09-30-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
Maybe useful as a starting point: SGLang on dual 3090, keeping token cost in mind from the start.
Feel free to tear it apart if you see something dumb.
My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.
Re: serving on Blackwell - sysstation69 - 10-02-2026
I hit almost the same issue last night.
Re: serving on Blackwell - torchtrack24 - 10-06-2026
My first guess would be this: If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious.
Re: serving on Blackwell - rasterbench - 10-06-2026
Interesting, I hadn't tried that. The pattern around HBM constraints has been pretty similar for me.
Re: serving on Blackwell - macrotinker - 10-06-2026
Appreciate the detail. That matches what I saw on A6000. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
If you post your exact settings, people here can probably narrow it down pretty fast.
Re: serving on Blackwell - syshouse - 10-09-2026
My first guess would be this: On M3 Max, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Re: serving on Blackwell - macrolab - 10-09-2026
I'd start with the simpler stack.
Re: serving on Blackwell - papermixer4 - 10-10-2026
A couple things I'd check first. If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious.
Re: serving on Blackwell - batchcraft - 10-10-2026
Thanks, that helps. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: serving on Blackwell - kernelmike - 10-10-2026
Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. I expected the model choice to matter more than it did.