+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Drivers / Frameworks / Toolkits (https://www.datasculpting.ai/inference/forumdisplay.php?fid=8)
+--- Forum: NVIDIA / CUDA (https://www.datasculpting.ai/inference/forumdisplay.php?fid=31)
+--- Thread: Blackwell deployment chatter on Threadripper box (/showthread.php?tid=1365)
Pages:
1
2
Blackwell deployment chatter on Threadripper box - audiopath66 - 10-17-2026
Posting this in case it saves someone else a little time.
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about KV cache pressure.
Would rather learn from your scars than collect my own all at once.
I care more about stability than squeezing the last possible tokens/sec out of it.
Re: Blackwell deployment chatter on Threadripper box - kernelmike - 10-17-2026
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at small local stacks. Once you know the plain path works, then add the faster path back in piece by piece.
Re: Blackwell deployment chatter on Threadripper box - audioriver - 10-17-2026
Can you fill in a bit more context? Is this local-only or are you routing anything externally?
Re: Blackwell deployment chatter on Threadripper box - orbitbench6 - 10-17-2026
Good point. The pattern around router policy has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: Blackwell deployment chatter on Threadripper box - kernelmike - 10-18-2026
I hit almost the same issue last night. I keep promising myself this is the last rebuild.
Re: Blackwell deployment chatter on Threadripper box - kernelmike - 10-18-2026
Quote:I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.
Maybe, but I had the opposite result. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Could be a workload difference, but I wouldn't generalize too fast from one setup.
Re: Blackwell deployment chatter on Threadripper box - mangohouse - 10-18-2026
Can you fill in a bit more context? Are you testing this on B200 or something else?
Re: Blackwell deployment chatter on Threadripper box - amberpatch25 - 10-18-2026
Not sure I agree there. I get why that seems right, but on 5090 the bottleneck felt elsewhere. Could be a workload difference, but I wouldn't generalize too fast from one setup.
Re: Blackwell deployment chatter on Threadripper box - amberpatch25 - 10-18-2026
A couple things I'd check first. On M3 Max, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece. Turns out 'just one more test' was a lie.
Re: Blackwell deployment chatter on Threadripper box - oxidemixer - 10-18-2026
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tokens/sec. Not glamorous advice, but repeatable baselines save a lot of time.