+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Drivers / Frameworks / Toolkits (https://www.datasculpting.ai/inference/forumdisplay.php?fid=8)
+--- Forum: NVIDIA / CUDA (https://www.datasculpting.ai/inference/forumdisplay.php?fid=31)
+--- Thread: Tensor cores actually helping here on B200 (/showthread.php?tid=673)
Pages:
1
2
Tensor cores actually helping here on B200 - dockember - 10-02-2026
Did a bit of testing and thought I'd post the result here.
After a few runs on 7900 XTX, I ended up with decent performance, but the setup only really made sense once I adjusted around MoE serving.
Happy to post more details if anyone wants them.
Re: Tensor cores actually helping here on B200 - amberpatch25 - 10-02-2026
That matches what I was seeing. I ended up in about the same place using LM Studio. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: Tensor cores actually helping here on B200 - phaseport36 - 10-03-2026
That matches what I was seeing. The pattern around reranking has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: Tensor cores actually helping here on B200 - kernelmike - 10-04-2026
Interesting, I hadn't tried that. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: Tensor cores actually helping here on B200 - thetapath - 10-04-2026
Interesting, I hadn't tried that. I ended up in about the same place using SGLang.
Re: Tensor cores actually helping here on B200 - latentgarden - 10-04-2026
What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.
Re: Tensor cores actually helping here on B200 - kernelmike - 10-05-2026
I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tokens/sec. Not glamorous advice, but repeatable baselines save a lot of time.
Re: Tensor cores actually helping here on B200 - nanobrook - 10-06-2026
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at small local stacks. Not glamorous advice, but repeatable baselines save a lot of time.
Re: Tensor cores actually helping here on B200 - dockember - 10-06-2026
Quote:A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at small local stacks. Not gl
I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at reranking.
Re: Tensor cores actually helping here on B200 - tinycontext - 10-06-2026
What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at long context. Not glamorous advice, but repeatable baselines save a lot of time.
I'm happy to test one more variant if you want a second data point.