+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Evaluation & Testing (https://www.datasculpting.ai/inference/forumdisplay.php?fid=16)
+--- Forum: Model Comparisons (https://www.datasculpting.ai/inference/forumdisplay.php?fid=67)
+--- Thread: flux pipeline local tips on B200 (/showthread.php?tid=74)
Pages:
1
2
flux pipeline local tips on B200 - cudaworks - 08-30-2026
This keeps coming up in conversation so I figured I'd start a thread.
The more I work with local setups, the more it seems like speculative decoding decides everything downstream.
Maybe obvious to some of you, but it clicked for me this week.
My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.
Re: flux pipeline local tips on B200 - routearch - 08-31-2026
Quote:This keeps coming up in conversation so I figured I'd start a thread.
Ran a quick check on my box. On 7900 XTX with LM Studio, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.
Re: flux pipeline local tips on B200 - southspark - 08-31-2026
That matches what I was seeing. The pattern around continuous batching has been pretty similar for me.
Re: flux pipeline local tips on B200 - rocmplane - 08-31-2026
My first guess would be this: If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.
Re: flux pipeline local tips on B200 - llmarch83 - 08-31-2026
Appreciate the detail. That matches what I saw on 7900 XTX. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: flux pipeline local tips on B200 - promptpoint - 09-01-2026
Not sure I agree there. I get why that seems right, but on A6000 the bottleneck felt elsewhere. Worth testing the simpler explanation before adopting the more interesting one.
Re: flux pipeline local tips on B200 - visionforge - 09-01-2026
I tested this tonight. On M3 Max with SGLang, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.
Re: flux pipeline local tips on B200 - cacheplane98 - 09-04-2026
What exact setup are you running? Are you testing this on Threadripper box or something else?
Re: flux pipeline local tips on B200 - llmtrack26 - 09-04-2026
I don't think that's the whole story. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the simpler explanation before adopting the more interesting one.
Re: flux pipeline local tips on B200 - macropoint - 09-05-2026
Not sure I agree there. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.